agora inbox for [email protected]
help / color / mirror / Atom feedRe: Disallowing multiple queries per PQexec()
279+ messages / 7 participants
[nested] [flat]
* Re: Disallowing multiple queries per PQexec()
@ 2017-03-01 22:02 Jim Nasby <[email protected]>
2017-03-02 09:01 ` Re: Disallowing multiple queries per PQexec() Surafel Temesgen <[email protected]>
0 siblings, 1 reply; 279+ messages in thread
From: Jim Nasby @ 2017-03-01 22:02 UTC (permalink / raw)
To: Andres Freund <[email protected]>; Andreas Karlsson <[email protected]>; +Cc: Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Surafel Temesgen <[email protected]>; pgsql-hackers
On 2/28/17 2:45 PM, Andres Freund wrote:
> So if you don't want to allow multiple statements, use PQexecParams et
> al.
That does leave most application authors out in the cold though, since
they're using a higher level connection manager.
If the maintenance burden isn't terribly high it would be nice to allow
disabling multiple statements via a GUC.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 279+ messages in thread
* Re: Disallowing multiple queries per PQexec()
2017-03-01 22:02 Re: Disallowing multiple queries per PQexec() Jim Nasby <[email protected]>
@ 2017-03-02 09:01 ` Surafel Temesgen <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Surafel Temesgen @ 2017-03-02 09:01 UTC (permalink / raw)
To: Jim Nasby <[email protected]>; +Cc: Andres Freund <[email protected]>; Andreas Karlsson <[email protected]>; Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers
As far as my understanding the issue at that time was inability to process
creation
of a database and connecting to it with one query string and that can be
solved by
fixing transaction restriction checks for CREATE DATABASE or disallowing
multiple
queries in PQexe.
If the issue solved and allowing multiple queries in PQexec doesn’t result
in SQL injection
attacks that worth backwards-compatibility breakage by itself the item can
be drop or
included to v4 Protocol section if it contains items that break
backwards-compatibility already
regards
surafel
On Thu, Mar 2, 2017 at 1:02 AM, Jim Nasby <[email protected]> wrote:
> On 2/28/17 2:45 PM, Andres Freund wrote:
>
>> So if you don't want to allow multiple statements, use PQexecParams et
>> al.
>>
>
> That does leave most application authors out in the cold though, since
> they're using a higher level connection manager.
>
> If the maintenance burden isn't terribly high it would be nice to allow
> disabling multiple statements via a GUC.
> --
> Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
> Experts in Analytics, Data Architecture and PostgreSQL
> Data in Trouble? Get it in Treble! http://BlueTreble.com
> 855-TREBLE2 (855-873-2532)
>
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH 3/5] Make archiver process an auxiliary process
@ 2018-11-07 07:53 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2018-11-07 07:53 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data wes moved onto shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/bootstrap/bootstrap.c | 8 +++
src/backend/postmaster/pgarch.c | 98 +++++++----------------------
src/backend/postmaster/pgstat.c | 6 ++
src/backend/postmaster/postmaster.c | 43 ++++++++-----
src/include/miscadmin.h | 2 +
src/include/pgstat.h | 1 +
src/include/postmaster/pgarch.h | 4 +-
7 files changed, 70 insertions(+), 92 deletions(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index bfc629c753..2f38420313 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -326,6 +326,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
statmsg = pgstat_get_backend_desc(B_STARTUP);
break;
+ case ArchiverProcess:
+ statmsg = pgstat_get_backend_desc(B_ARCHIVER);
+ break;
case BgWriterProcess:
statmsg = pgstat_get_backend_desc(B_BG_WRITER);
break;
@@ -451,6 +454,11 @@ AuxiliaryProcessMain(int argc, char *argv[])
StartupProcessMain();
proc_exit(1); /* should never return */
+ case ArchiverProcess:
+ /* don't set signals, archiver has its own agenda */
+ PgArchiverMain();
+ proc_exit(1); /* should never return */
+
case BgWriterProcess:
/* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 3ca30badb2..6441e69e9a 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -78,7 +78,6 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
@@ -95,7 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
static void pgarch_exit(SIGNAL_ARGS);
static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
@@ -111,75 +109,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -219,8 +148,8 @@ pgarch_forkexec(void)
* The argc/argv parameters are valid only in EXEC_BACKEND case. However,
* since we don't use 'em, it hardly matters...
*/
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -252,8 +181,27 @@ PgArchiverMain(int argc, char *argv[])
static void
pgarch_exit(SIGNAL_ARGS)
{
- /* SIGQUIT means curl up and die ... */
- exit(1);
+ PG_SETMASK(&BlockSig);
+
+ /*
+ * We DO NOT want to run proc_exit() callbacks -- we're here because
+ * shared memory may be corrupted, so we don't want to try to clean up our
+ * transaction. Just nail the windows shut and get out of town. Now that
+ * there's an atexit callback to prevent third-party code from breaking
+ * things by calling exit() directly, we have to reset the callbacks
+ * explicitly to make this work as intended.
+ */
+ on_exit_reset();
+
+ /*
+ * Note we do exit(2) not exit(0). This is to force the postmaster into a
+ * system reset cycle if some idiot DBA sends a manual SIGQUIT to a random
+ * process. This is necessary precisely because we don't clean up our
+ * shared memory state. (The "dead man switch" mechanism in pmsignal.c
+ * should ensure the postmaster sees this as a crash, too, but no harm in
+ * being doubly sure.)
+ */
+ exit(2);
}
/* SIGUSR1 signal handler for archiver process */
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 51c486bebd..ca5c6376e5 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -2927,6 +2927,9 @@ pgstat_bestart(void)
case StartupProcess:
lbeentry.st_backendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ beentry->st_backendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
lbeentry.st_backendType = B_BG_WRITER;
break;
@@ -4273,6 +4276,9 @@ pgstat_get_backend_desc(BackendType backendType)
switch (backendType)
{
+ case B_ARCHIVER:
+ backendDesc = "archiver";
+ break;
case B_AUTOVAC_LAUNCHER:
backendDesc = "autovacuum launcher";
break;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7a92dac525..ee41d7009e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -146,7 +146,8 @@
#define BACKEND_TYPE_AUTOVAC 0x0002 /* autovacuum worker process */
#define BACKEND_TYPE_WALSND 0x0004 /* walsender process */
#define BACKEND_TYPE_BGWORKER 0x0008 /* bgworker process */
-#define BACKEND_TYPE_ALL 0x000F /* OR of all the above */
+#define BACKEND_TYPE_ARCHIVER 0x0010 /* archiver process */
+#define BACKEND_TYPE_ALL 0x001F /* OR of all the above */
#define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER)
@@ -539,6 +540,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1785,7 +1787,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3042,7 +3044,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3177,20 +3179,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3438,7 +3436,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3643,6 +3641,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3915,6 +3925,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5190,7 +5201,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5475,6 +5486,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 62d64aa0a1..dc7dd1c164 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -397,6 +397,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -409,6 +410,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 36b530bc27..0334213b98 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -718,6 +718,7 @@ typedef struct PgStat_GlobalStats
*/
typedef enum BackendType
{
+ B_ARCHIVER,
B_AUTOVAC_LAUNCHER,
B_AUTOVAC_WORKER,
B_BACKEND,
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
--
2.23.0
----Next_Part(Wed_Jan_22_17_24_04_2020_620)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v24 3/5] Make archiver process an auxiliary process
@ 2018-11-07 07:53 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2018-11-07 07:53 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data wes moved onto shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/bootstrap/bootstrap.c | 8 +++
src/backend/postmaster/pgarch.c | 98 +++++++----------------------
src/backend/postmaster/pgstat.c | 6 ++
src/backend/postmaster/postmaster.c | 35 ++++++++---
src/include/miscadmin.h | 2 +
src/include/pgstat.h | 1 +
src/include/postmaster/pgarch.h | 4 +-
7 files changed, 67 insertions(+), 87 deletions(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 8ea033610d..6e38f9a3d2 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -329,6 +329,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case BgWriterProcess:
statmsg = pgstat_get_backend_desc(B_BG_WRITER);
break;
+ case ArchiverProcess:
+ statmsg = pgstat_get_backend_desc(B_ARCHIVER);
+ break;
case CheckpointerProcess:
statmsg = pgstat_get_backend_desc(B_CHECKPOINTER);
break;
@@ -456,6 +459,11 @@ AuxiliaryProcessMain(int argc, char *argv[])
BackgroundWriterMain();
proc_exit(1); /* should never return */
+ case ArchiverProcess:
+ /* don't set signals, archiver has its own agenda */
+ PgArchiverMain();
+ proc_exit(1); /* should never return */
+
case CheckpointerProcess:
/* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index f84f882c4c..4342ebdab4 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -77,7 +77,6 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
@@ -96,7 +95,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
static void pgarch_exit(SIGNAL_ARGS);
static void ArchSigHupHandler(SIGNAL_ARGS);
static void ArchSigTermHandler(SIGNAL_ARGS);
@@ -114,75 +112,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -222,8 +151,8 @@ pgarch_forkexec(void)
* The argc/argv parameters are valid only in EXEC_BACKEND case. However,
* since we don't use 'em, it hardly matters...
*/
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -255,8 +184,27 @@ PgArchiverMain(int argc, char *argv[])
static void
pgarch_exit(SIGNAL_ARGS)
{
- /* SIGQUIT means curl up and die ... */
- exit(1);
+ PG_SETMASK(&BlockSig);
+
+ /*
+ * We DO NOT want to run proc_exit() callbacks -- we're here because
+ * shared memory may be corrupted, so we don't want to try to clean up our
+ * transaction. Just nail the windows shut and get out of town. Now that
+ * there's an atexit callback to prevent third-party code from breaking
+ * things by calling exit() directly, we have to reset the callbacks
+ * explicitly to make this work as intended.
+ */
+ on_exit_reset();
+
+ /*
+ * Note we do exit(2) not exit(0). This is to force the postmaster into a
+ * system reset cycle if some idiot DBA sends a manual SIGQUIT to a random
+ * backend. This is necessary precisely because we don't clean up our
+ * shared memory state. (The "dead man switch" mechanism in pmsignal.c
+ * should ensure the postmaster sees this as a crash, too, but no harm in
+ * being doubly sure.)
+ */
+ exit(2);
}
/* SIGHUP signal handler for archiver process */
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index fabcf31de8..8299d2a435 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -2932,6 +2932,9 @@ pgstat_bestart(void)
case StartupProcess:
lbeentry.st_backendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ beentry->st_backendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
lbeentry.st_backendType = B_BG_WRITER;
break;
@@ -4275,6 +4278,9 @@ pgstat_get_backend_desc(BackendType backendType)
switch (backendType)
{
+ case B_ARCHIVER:
+ backendDesc = "archiver";
+ break;
case B_AUTOVAC_LAUNCHER:
backendDesc = "autovacuum launcher";
break;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 9ff2832c00..84fda38249 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -146,7 +146,8 @@
#define BACKEND_TYPE_AUTOVAC 0x0002 /* autovacuum worker process */
#define BACKEND_TYPE_WALSND 0x0004 /* walsender process */
#define BACKEND_TYPE_BGWORKER 0x0008 /* bgworker process */
-#define BACKEND_TYPE_ALL 0x000F /* OR of all the above */
+#define BACKEND_TYPE_ARCHIVER 0x0010 /* archiver process */
+#define BACKEND_TYPE_ALL 0x001F /* OR of all the above */
#define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER)
@@ -539,6 +540,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1785,7 +1787,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3042,7 +3044,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3187,10 +3189,8 @@ reaper(SIGNAL_ARGS)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3438,7 +3438,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3643,6 +3643,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3915,6 +3927,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5190,7 +5203,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5475,6 +5488,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index bc6e03fbc7..1f4db67f3f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -399,6 +399,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -411,6 +412,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fe076d823d..65713abc2b 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -718,6 +718,7 @@ typedef struct PgStat_GlobalStats
*/
typedef enum BackendType
{
+ B_ARCHIVER,
B_AUTOVAC_LAUNCHER,
B_AUTOVAC_WORKER,
B_BACKEND,
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index 2474eac26a..88f16863d4 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
--
2.23.0
----Next_Part(Tue_Dec__3_17_27_59_2019_887)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v24-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v32 4/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index d62c12310a..2f8672ac0c 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -489,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 5480a024e0..d398ce6f03 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -319,6 +319,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -439,30 +442,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 37be0e2bbb..063d1323ea 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
pqsignal(SIGQUIT, SignalHandlerForCrashExit);
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 73d278f3b2..e97995f973 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -539,6 +539,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1785,7 +1786,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3055,7 +3056,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3190,20 +3191,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3451,7 +3448,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3656,6 +3653,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3928,6 +3937,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5208,7 +5218,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5251,16 +5261,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5493,6 +5493,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 9938cddb57..af4599bd82 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -187,6 +187,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 9ec7b31cce..4435df82b6 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -313,6 +313,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 14fa127ab1..619b2f9c71 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index d21780108b..a87e7dc711 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -272,6 +272,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.18.2
----Next_Part(Tue_Apr__7_16_38_17_2020_299)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v32-0005-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v44 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index f39dc4ddf1..d69e20961d 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index a7ed93fdc1..edda899554 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ed1b65358d..e3a520def9 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 5d09822c81..8a4706a870 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -547,6 +547,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1787,7 +1788,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3041,7 +3042,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3176,20 +3177,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3437,7 +3434,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3642,6 +3639,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3938,6 +3947,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5185,7 +5195,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5230,16 +5240,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5481,6 +5481,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 7dc3911590..fc23539137 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -181,6 +181,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 221af87e71..65443063e8 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..bba002ad24 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -418,6 +418,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -430,6 +431,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index e77f76ae8a..a656910d02 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Mon_Dec_21_17_16_20_2020_608)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v44-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v37 3/6] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 8f8734dc1d..2e6c322142 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -489,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 76b2f5066f..81bfaea869 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ed1b65358d..e3a520def9 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 959e3b8873..b811c961a6 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -555,6 +555,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1800,7 +1801,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3054,7 +3055,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3189,20 +3190,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3450,7 +3447,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3655,6 +3652,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3951,6 +3960,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5230,7 +5240,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5275,16 +5285,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5526,6 +5526,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 88566bd9fa..746bed773e 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -181,6 +181,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 221af87e71..65443063e8 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..bba002ad24 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -418,6 +418,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -430,6 +431,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 9c9a50ae45..de20520b8c 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -345,6 +345,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.18.4
----Next_Part(Fri_Sep_25_09_27_26_2020_419)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v37-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v42 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index cae93ab69d..6908bec2f9 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -489,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index a7ed93fdc1..edda899554 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ed1b65358d..e3a520def9 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index b7799ed1d2..a1dd6964e9 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -547,6 +547,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1785,7 +1786,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3039,7 +3040,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3174,20 +3175,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3435,7 +3432,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3640,6 +3637,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3936,6 +3945,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5177,7 +5187,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5222,16 +5232,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5473,6 +5473,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 88566bd9fa..746bed773e 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -181,6 +181,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 221af87e71..65443063e8 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..bba002ad24 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -418,6 +418,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -430,6 +431,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 9c9a50ae45..de20520b8c 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -345,6 +345,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.18.4
----Next_Part(Wed_Nov_11_10_07_22_2020_796)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v42-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v41 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index cae93ab69d..6908bec2f9 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -489,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index a7ed93fdc1..edda899554 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ed1b65358d..e3a520def9 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 959e3b8873..b811c961a6 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -555,6 +555,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1800,7 +1801,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3054,7 +3055,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3189,20 +3190,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3450,7 +3447,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3655,6 +3652,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3951,6 +3960,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5230,7 +5240,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5275,16 +5285,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5526,6 +5526,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 88566bd9fa..746bed773e 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -181,6 +181,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 221af87e71..65443063e8 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..bba002ad24 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -418,6 +418,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -430,6 +431,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 9c9a50ae45..de20520b8c 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -345,6 +345,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.18.4
----Next_Part(Fri_Nov__6_09_27_56_2020_738)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v41-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v43 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index f39dc4ddf1..d69e20961d 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index a7ed93fdc1..edda899554 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ed1b65358d..e3a520def9 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 5d09822c81..8a4706a870 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -547,6 +547,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1787,7 +1788,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3041,7 +3042,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3176,20 +3177,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3437,7 +3434,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3642,6 +3639,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3938,6 +3947,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5185,7 +5195,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5230,16 +5240,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5481,6 +5481,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 7dc3911590..fc23539137 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -181,6 +181,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 221af87e71..65443063e8 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..bba002ad24 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -418,6 +418,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -430,6 +431,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index e77f76ae8a..a656910d02 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Dec_11_16_50_03_2020_915)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v43-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v33 4/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 55becd65d4..2417de72a0 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -489,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 5480a024e0..d398ce6f03 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -319,6 +319,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -439,30 +442,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 37be0e2bbb..063d1323ea 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
pqsignal(SIGQUIT, SignalHandlerForCrashExit);
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 0578e92ba9..2687d2af47 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -539,6 +539,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1785,7 +1786,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3055,7 +3056,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3190,20 +3191,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3451,7 +3448,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3656,6 +3653,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3928,6 +3937,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5203,7 +5213,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5246,16 +5256,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5488,6 +5488,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 5aa19d3f78..7acc48734e 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -187,6 +187,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 0a12afb59e..099da8cd76 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -336,6 +336,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 14fa127ab1..619b2f9c71 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index ae4f573ab4..1a8a0c2e15 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -272,6 +272,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.18.2
----Next_Part(Thu_Apr_30_13_22_05_2020_409)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v33-0005-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v29 4/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlog.c | 46 ++++++++++
src/backend/access/transam/xlogarchive.c | 2 +-
src/backend/bootstrap/bootstrap.c | 22 ++---
src/backend/postmaster/pgarch.c | 102 ++++-------------------
src/backend/postmaster/postmaster.c | 53 ++++++------
src/include/access/xlog.h | 2 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
10 files changed, 108 insertions(+), 127 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 977d448f50..d75a257da2 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -684,6 +684,13 @@ typedef struct XLogCtlData
*/
Latch recoveryWakeupLatch;
+ /*
+ * archiverWakeupLatch is used to wake up the archiver process to process
+ * completed WAL segments, if it is waiting for WAL to arrive. Protected
+ * by info_lck.
+ */
+ Latch *archiverWakeupLatch;
+
/*
* During recovery, we keep a copy of the latest checkpoint record here.
* lastCheckPointRecPtr points to start of checkpoint record and
@@ -8450,6 +8457,45 @@ GetLastSegSwitchData(XLogRecPtr *lastSwitchLSN)
return result;
}
+/*
+ * XLogArchiveWakeupStart - Set up archiver wakeup stuff
+ */
+void
+XLogArchiveWakeupStart(void)
+{
+ SpinLockAcquire(&XLogCtl->info_lck);
+ Assert(XLogCtl->archiverWakeupLatch == NULL);
+ XLogCtl->archiverWakeupLatch = MyLatch;
+ SpinLockRelease(&XLogCtl->info_lck);
+}
+
+/*
+ * XLogArchiveWakeupEnd - Clean up archiver wakeup stuff
+ */
+void
+XLogArchiveWakeupEnd(void)
+{
+ SpinLockAcquire(&XLogCtl->info_lck);
+ XLogCtl->archiverWakeupLatch = NULL;
+ SpinLockRelease(&XLogCtl->info_lck);
+}
+
+/*
+ * XLogWakeupArchiver - Wake up archiver process
+ */
+void
+XLogArchiveWakeup(void)
+{
+ Latch *latch;
+
+ SpinLockAcquire(&XLogCtl->info_lck);
+ latch = XLogCtl->archiverWakeupLatch;
+ SpinLockRelease(&XLogCtl->info_lck);
+
+ if (latch)
+ SetLatch(latch);
+}
+
/*
* This must be called ONCE during postmaster or standalone-backend shutdown
*/
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index d62c12310a..d6c8bcce8b 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -490,7 +490,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ XLogArchiveWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 5480a024e0..d398ce6f03 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -319,6 +319,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -439,30 +442,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 37be0e2bbb..6fe7a136ba 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,7 +79,6 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
@@ -95,8 +95,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +108,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +141,21 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
+/* Clean up notification stuff on exit */
+static void
+PgArchiverKill(int code, Datum arg)
+{
+ XLogArchiveWakeupEnd();
+}
+
/*
* PgArchiverMain
*
* The argc/argv parameters are valid only in EXEC_BACKEND case. However,
* since we don't use 'em, it hardly matters...
*/
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,7 +167,7 @@ PgArchiverMain(int argc, char *argv[])
pqsignal(SIGQUIT, SignalHandlerForCrashExit);
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
@@ -240,24 +176,14 @@ PgArchiverMain(int argc, char *argv[])
MyBackendType = B_ARCHIVER;
init_ps_display(NULL);
+ XLogArchiveWakeupStart();
+ on_shmem_exit(PgArchiverKill, 0);
+
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 73d278f3b2..a4b9d212a2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -146,7 +146,8 @@
#define BACKEND_TYPE_AUTOVAC 0x0002 /* autovacuum worker process */
#define BACKEND_TYPE_WALSND 0x0004 /* walsender process */
#define BACKEND_TYPE_BGWORKER 0x0008 /* bgworker process */
-#define BACKEND_TYPE_ALL 0x000F /* OR of all the above */
+#define BACKEND_TYPE_ARCHIVER 0x0010 /* archiver process */
+#define BACKEND_TYPE_ALL 0x001F /* OR of all the above */
#define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER)
@@ -539,6 +540,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1785,7 +1787,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3055,7 +3057,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3190,20 +3192,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3451,7 +3449,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3656,6 +3654,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3928,6 +3938,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5208,7 +5219,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5251,16 +5262,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5493,6 +5494,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 9ec7b31cce..c0e149d526 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -313,6 +313,8 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 14fa127ab1..619b2f9c71 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
--
2.18.2
----Next_Part(Wed_Apr__1_15_15_11_2020_923)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v29-0005-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v36 4/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 8f8734dc1d..2e6c322142 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -489,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 76b2f5066f..81bfaea869 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 37be0e2bbb..063d1323ea 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
pqsignal(SIGQUIT, SignalHandlerForCrashExit);
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 42223c0f61..4c296dd214 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -554,6 +554,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1799,7 +1800,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3051,7 +3052,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3186,20 +3187,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3447,7 +3444,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3652,6 +3649,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3948,6 +3957,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5223,7 +5233,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5268,16 +5278,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5515,6 +5515,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index aa9fbd8054..9cc5226667 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -181,6 +181,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 221af87e71..65443063e8 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..bba002ad24 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -418,6 +418,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -430,6 +431,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 9c9a50ae45..de20520b8c 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -345,6 +345,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.18.4
----Next_Part(Tue_Sep__8_17_55_57_2020_197)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v36-0005-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v25 4/8] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/bootstrap/bootstrap.c | 22 +++++----
src/backend/postmaster/pgarch.c | 75 +----------------------------
src/backend/postmaster/postmaster.c | 43 +++++++++++------
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
5 files changed, 46 insertions(+), 100 deletions(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 5480a024e0..d398ce6f03 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -319,6 +319,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -439,30 +442,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 37be0e2bbb..4971b3ae42 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -78,7 +78,6 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
@@ -95,7 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
@@ -110,75 +108,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -218,8 +147,8 @@ pgarch_forkexec(void)
* The argc/argv parameters are valid only in EXEC_BACKEND case. However,
* since we don't use 'em, it hardly matters...
*/
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 2b9ab32293..cab7fb5381 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -146,7 +146,8 @@
#define BACKEND_TYPE_AUTOVAC 0x0002 /* autovacuum worker process */
#define BACKEND_TYPE_WALSND 0x0004 /* walsender process */
#define BACKEND_TYPE_BGWORKER 0x0008 /* bgworker process */
-#define BACKEND_TYPE_ALL 0x000F /* OR of all the above */
+#define BACKEND_TYPE_ARCHIVER 0x0010 /* archiver process */
+#define BACKEND_TYPE_ALL 0x001F /* OR of all the above */
#define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER)
@@ -539,6 +540,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1785,7 +1787,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3055,7 +3057,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3190,20 +3192,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3451,7 +3449,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3656,6 +3654,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3928,6 +3938,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5208,7 +5219,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5493,6 +5504,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 14fa127ab1..619b2f9c71 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
--
2.18.2
----Next_Part(Thu_Mar_19_20_30_04_2020_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v25-0005-Use-latch-instead-of-SIGUSR1-to-wake-up-archiver.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v35 4/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index cdd586fcfb..ee3444284b 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -489,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 5480a024e0..d398ce6f03 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -319,6 +319,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -439,30 +442,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 37be0e2bbb..063d1323ea 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
pqsignal(SIGQUIT, SignalHandlerForCrashExit);
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index b4d475bb0b..30f3fc6ba1 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -539,6 +539,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1785,7 +1786,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3068,7 +3069,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3203,20 +3204,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3464,7 +3461,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3669,6 +3666,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3941,6 +3950,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5216,7 +5226,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5259,16 +5269,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5501,6 +5501,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index f5eef6fa4e..46f0198510 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -187,6 +187,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index e917dfe92d..0783692c83 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -336,6 +336,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 18bc8a7b90..cf0b463eba 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 1ee9000b2b..8ab6d859bb 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -270,6 +270,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.18.2
----Next_Part(Mon_Jun__8_17_32_04_2020_489)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v35-0005-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v31 4/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index d62c12310a..2f8672ac0c 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -489,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 5480a024e0..d398ce6f03 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -319,6 +319,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -439,30 +442,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 37be0e2bbb..063d1323ea 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
pqsignal(SIGQUIT, SignalHandlerForCrashExit);
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 73d278f3b2..e97995f973 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -539,6 +539,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1785,7 +1786,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3055,7 +3056,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3190,20 +3191,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3451,7 +3448,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3656,6 +3653,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3928,6 +3937,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5208,7 +5218,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5251,16 +5261,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5493,6 +5493,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 9938cddb57..af4599bd82 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -187,6 +187,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 9ec7b31cce..4435df82b6 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -313,6 +313,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 14fa127ab1..619b2f9c71 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index d21780108b..a87e7dc711 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -272,6 +272,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.18.2
----Next_Part(Fri_Apr__3_17_31_17_2020_104)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v31-0005-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v40 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index cae93ab69d..6908bec2f9 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -489,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index a7ed93fdc1..edda899554 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ed1b65358d..e3a520def9 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 959e3b8873..b811c961a6 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -555,6 +555,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1800,7 +1801,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3054,7 +3055,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3189,20 +3190,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3450,7 +3447,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3655,6 +3652,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3951,6 +3960,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5230,7 +5240,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5275,16 +5285,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5526,6 +5526,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 88566bd9fa..746bed773e 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -181,6 +181,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 221af87e71..65443063e8 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..bba002ad24 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -418,6 +418,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -430,6 +431,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 9c9a50ae45..de20520b8c 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -345,6 +345,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.18.4
----Next_Part(Wed_Nov__4_17_39_10_2020_208)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v40-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v39 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index cae93ab69d..6908bec2f9 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -489,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 76b2f5066f..81bfaea869 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ed1b65358d..e3a520def9 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 959e3b8873..b811c961a6 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -555,6 +555,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1800,7 +1801,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3054,7 +3055,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3189,20 +3190,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3450,7 +3447,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3655,6 +3652,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3951,6 +3960,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5230,7 +5240,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5275,16 +5285,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5526,6 +5526,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 88566bd9fa..746bed773e 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -181,6 +181,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 221af87e71..65443063e8 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..bba002ad24 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -418,6 +418,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -430,6 +431,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 9c9a50ae45..de20520b8c 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -345,6 +345,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.18.4
----Next_Part(Tue_Oct__6_10_06_44_2020_382)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v39-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v38 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index cae93ab69d..6908bec2f9 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -489,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 76b2f5066f..81bfaea869 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ed1b65358d..e3a520def9 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 959e3b8873..b811c961a6 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -555,6 +555,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1800,7 +1801,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3054,7 +3055,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3189,20 +3190,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3450,7 +3447,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3655,6 +3652,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3951,6 +3960,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5230,7 +5240,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5275,16 +5285,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5526,6 +5526,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 88566bd9fa..746bed773e 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -181,6 +181,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 221af87e71..65443063e8 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..bba002ad24 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -418,6 +418,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -430,6 +431,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 9c9a50ae45..de20520b8c 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -345,6 +345,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.18.4
----Next_Part(Thu_Oct__1_09_07_22_2020_252)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v38-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v34 4/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index cdd586fcfb..ee3444284b 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -489,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 5480a024e0..d398ce6f03 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -319,6 +319,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -439,30 +442,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 37be0e2bbb..063d1323ea 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
pqsignal(SIGQUIT, SignalHandlerForCrashExit);
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 160afe9f39..9de9396628 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -539,6 +539,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1785,7 +1786,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3068,7 +3069,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3203,20 +3204,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3464,7 +3461,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3669,6 +3666,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3941,6 +3950,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5216,7 +5226,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5259,16 +5269,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5501,6 +5501,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index f5eef6fa4e..46f0198510 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -187,6 +187,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index e917dfe92d..0783692c83 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -336,6 +336,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 14fa127ab1..619b2f9c71 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 1ee9000b2b..8ab6d859bb 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -270,6 +270,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.18.2
----Next_Part(Mon_Jun__1_18_00_01_2020_089)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v34-0005-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v49 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_16_53_11_2021_575)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v49-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v52 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 224 ++++++++---------------
src/backend/postmaster/postmaster.c | 83 ++++-----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
9 files changed, 139 insertions(+), 222 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..2558bcfb08 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,13 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +493,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c399e6c267 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,68 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver ", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +144,48 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +210,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +248,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +270,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +656,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..34c8551288 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_17_51_37_2021_192)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v52-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v33 4/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index cdd586fcfb..ee3444284b 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -489,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 5480a024e0..d398ce6f03 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -319,6 +319,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -439,30 +442,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 37be0e2bbb..063d1323ea 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
pqsignal(SIGQUIT, SignalHandlerForCrashExit);
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 160afe9f39..9de9396628 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -539,6 +539,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1785,7 +1786,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3068,7 +3069,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3203,20 +3204,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3464,7 +3461,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3669,6 +3666,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3941,6 +3950,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5216,7 +5226,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5259,16 +5269,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5501,6 +5501,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 5aa19d3f78..7acc48734e 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -187,6 +187,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index e917dfe92d..0783692c83 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -336,6 +336,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 14fa127ab1..619b2f9c71 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index ae4f573ab4..1a8a0c2e15 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -272,6 +272,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.18.2
----Next_Part(Fri_May_15_17_30_36_2020_111)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v33-0005-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v46 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_14_15_14_25_2021_903)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v46-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v45 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index db0cfaa360..aabf9d73eb 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0786fcf103..430d438303 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -354,6 +354,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v45-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v51 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 194 ++++-------------------
src/backend/postmaster/postmaster.c | 83 +++++-----
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 94 insertions(+), 236 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..c219780c60 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,13 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,148 +75,29 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +109,27 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +602,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ ProcGlobal->archiverLatch = NULL;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..fd509869aa 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3141,24 +3142,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3175,6 +3158,26 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving of remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
+ * PostmasterStateMachine() will advance to the next shutdown step.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ if (PgArchStartupAllowed())
+ PgArchPID = StartArchiver();
+ continue;
+ }
+
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d71700a565 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -350,6 +350,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Wed_Mar_10_12_10_39_2021_432)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v51-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v48 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..22e1cd244c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3403,7 +3400,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,6 +3606,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3912,6 +3921,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5140,7 +5150,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5204,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5445,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..491747c113 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Fri_Mar__5_17_18_56_2021_497)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v48-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v47 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 6 +-
src/backend/bootstrap/bootstrap.c | 22 ++--
src/backend/postmaster/pgarch.c | 130 +++--------------------
src/backend/postmaster/postmaster.c | 50 +++++----
src/backend/storage/lmgr/proc.c | 1 +
src/include/access/xlog.h | 3 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 3 +
11 files changed, 69 insertions(+), 154 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..d01859bde5 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,9 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/latch.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +492,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..9a2e21bf86 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,13 +79,11 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
@@ -95,8 +94,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +107,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +140,9 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +154,26 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +198,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +236,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +258,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7de27ee4e0..af91c313e2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1788,7 +1789,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3027,7 +3028,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3162,20 +3163,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3423,7 +3420,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3629,6 +3626,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3932,6 +3941,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5160,7 +5170,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5214,16 +5224,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5465,6 +5465,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index c87ffc6549..a1e51c5b99 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 75ec1073bd..551f518cc2 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -339,6 +339,9 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
+extern void XLogArchiveWakeup(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 3edd1a976c..1a59181cf9 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..385b002dfe 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 683ab64f76..6cdaf3753d 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -355,6 +355,9 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
+ /* Current shared estimate of appropriate spins_per_delay value */
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
--
2.27.0
----Next_Part(Thu_Jan_21_12_03_48_2021_284)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v47-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v50 3/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlogarchive.c | 7 +-
src/backend/bootstrap/bootstrap.c | 22 +--
src/backend/postmaster/pgarch.c | 174 +++--------------------
src/backend/postmaster/postmaster.c | 88 ++++++------
src/backend/storage/lmgr/proc.c | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 10 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 10 +-
9 files changed, 83 insertions(+), 232 deletions(-)
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..8b66a30931 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -29,7 +29,8 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -490,8 +491,8 @@ XLogArchiveNotify(const char *xlog)
}
/* Notify archiver that it's got something to do */
- if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ if (IsUnderPostmaster && ProcGlobal->archiverLatch)
+ SetLatch(ProcGlobal->archiverLatch);
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..e237cedaff 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,25 +79,17 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -105,121 +98,9 @@ static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
-}
-#endif /* EXEC_BACKEND */
-
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,31 +112,24 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->archiverLatch = &MyProc->procLatch;
pgarch_MainLoop();
- exit(0);
-}
-
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ proc_exit(0);
}
/* SIGUSR2 signal handler for archiver process */
@@ -282,14 +156,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +194,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +216,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..d9f8d82650 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3126,6 +3127,18 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the archiver? We treat it the same way to WAL receiver.
+ */
+ if (pid == PgArchPID)
+ {
+ PgArchPID = 0;
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3141,24 +3154,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
- */
- if (pid == PgArchPID)
- {
- PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
- continue;
- }
-
/*
* Was it the statistics collector? If so, just try to start a new
* one; no need to force reset of the rest of the system. (If fail,
@@ -3403,7 +3398,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3604,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3912,6 +3904,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -4989,6 +4982,19 @@ SubPostmasterMain(int argc, char *argv[])
AuxiliaryProcessMain(argc - 2, argv + 2); /* does not return */
}
+ if (strcmp(argv[1], "--forkarch") == 0)
+ {
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitAuxiliaryProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores();
+
+ PgArchiverMain(); /* does not return */
+ }
if (strcmp(argv[1], "--forkavlauncher") == 0)
{
/* Restore basic shared memory pointers */
@@ -5037,12 +5043,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5140,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5194,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5435,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 897045ee27..bf5d6da89b 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->archiverLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..adb9f819bb 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -419,6 +419,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -431,6 +432,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..737ed69e57 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,6 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..d73890b228 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
int startupProcPid;
/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
int startupBufferPinWaitBufId;
+ /* Archiver process's latch */
+ Latch *archiverLatch;
} PROC_HDR;
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
@@ -370,11 +372,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0004-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v30 4/7] Make archiver process an auxiliary process
@ 2020-03-13 07:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2020-03-13 07:59 UTC (permalink / raw)
This is a preliminary patch for shared-memory based stats collector.
Archiver process must be a auxiliary process since it uses shared
memory after stats data was moved into shared-memory. Make the process
an auxiliary process in order to make it work.
---
src/backend/access/transam/xlog.c | 46 +++++++++
src/backend/access/transam/xlogarchive.c | 2 +-
src/backend/bootstrap/bootstrap.c | 22 +++--
src/backend/postmaster/pgarch.c | 115 ++++-------------------
src/backend/postmaster/postmaster.c | 53 ++++++-----
src/include/access/xlog.h | 2 +
src/include/access/xlogarchive.h | 1 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 4 +-
src/include/storage/pmsignal.h | 1 -
10 files changed, 112 insertions(+), 136 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 977d448f50..d75a257da2 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -684,6 +684,13 @@ typedef struct XLogCtlData
*/
Latch recoveryWakeupLatch;
+ /*
+ * archiverWakeupLatch is used to wake up the archiver process to process
+ * completed WAL segments, if it is waiting for WAL to arrive. Protected
+ * by info_lck.
+ */
+ Latch *archiverWakeupLatch;
+
/*
* During recovery, we keep a copy of the latest checkpoint record here.
* lastCheckPointRecPtr points to start of checkpoint record and
@@ -8450,6 +8457,45 @@ GetLastSegSwitchData(XLogRecPtr *lastSwitchLSN)
return result;
}
+/*
+ * XLogArchiveWakeupStart - Set up archiver wakeup stuff
+ */
+void
+XLogArchiveWakeupStart(void)
+{
+ SpinLockAcquire(&XLogCtl->info_lck);
+ Assert(XLogCtl->archiverWakeupLatch == NULL);
+ XLogCtl->archiverWakeupLatch = MyLatch;
+ SpinLockRelease(&XLogCtl->info_lck);
+}
+
+/*
+ * XLogArchiveWakeupEnd - Clean up archiver wakeup stuff
+ */
+void
+XLogArchiveWakeupEnd(void)
+{
+ SpinLockAcquire(&XLogCtl->info_lck);
+ XLogCtl->archiverWakeupLatch = NULL;
+ SpinLockRelease(&XLogCtl->info_lck);
+}
+
+/*
+ * XLogWakeupArchiver - Wake up archiver process
+ */
+void
+XLogArchiveWakeup(void)
+{
+ Latch *latch;
+
+ SpinLockAcquire(&XLogCtl->info_lck);
+ latch = XLogCtl->archiverWakeupLatch;
+ SpinLockRelease(&XLogCtl->info_lck);
+
+ if (latch)
+ SetLatch(latch);
+}
+
/*
* This must be called ONCE during postmaster or standalone-backend shutdown
*/
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index d62c12310a..d6c8bcce8b 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -490,7 +490,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ XLogArchiveWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 5480a024e0..d398ce6f03 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -319,6 +319,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -439,30 +442,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 37be0e2bbb..7b66cb5613 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -48,6 +48,7 @@
#include "storage/latch.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -78,7 +79,6 @@
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
/*
@@ -95,8 +95,6 @@ static volatile sig_atomic_t ready_to_stop = false;
static pid_t pgarch_forkexec(void);
#endif
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
@@ -110,75 +108,6 @@ static void pgarch_archiveDone(char *xlog);
* ------------------------------------------------------------
*/
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
-
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
/* ------------------------------------------------------------
* Local functions called by archiver follow
* ------------------------------------------------------------
@@ -212,14 +141,16 @@ pgarch_forkexec(void)
#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Clean up notification stuff on exit */
+static void
+PgArchiverKill(int code, Datum arg)
+{
+ XLogArchiveWakeupEnd();
+}
+
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +162,23 @@ PgArchiverMain(int argc, char *argv[])
pqsignal(SIGQUIT, SignalHandlerForCrashExit);
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
-
+ XLogArchiveWakeupStart();
+ on_shmem_exit(PgArchiverKill, 0);
+
pgarch_MainLoop();
exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 73d278f3b2..a4b9d212a2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -146,7 +146,8 @@
#define BACKEND_TYPE_AUTOVAC 0x0002 /* autovacuum worker process */
#define BACKEND_TYPE_WALSND 0x0004 /* walsender process */
#define BACKEND_TYPE_BGWORKER 0x0008 /* bgworker process */
-#define BACKEND_TYPE_ALL 0x000F /* OR of all the above */
+#define BACKEND_TYPE_ARCHIVER 0x0010 /* archiver process */
+#define BACKEND_TYPE_ALL 0x001F /* OR of all the above */
#define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER)
@@ -539,6 +540,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1785,7 +1787,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3055,7 +3057,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3190,20 +3192,16 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
- * PostmasterStateMachine() will advance to the next shutdown step.
+ * Was it the archiver? Normal exit can be ignored; we'll start a new
+ * one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
- if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
continue;
}
@@ -3451,7 +3449,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3656,6 +3654,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) PgArchPID)));
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/*
* Force a power-cycle of the pgarch process too. (This isn't absolutely
* necessary, but it seems like a good idea for robustness, and it
@@ -3928,6 +3938,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5208,7 +5219,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5251,16 +5262,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5493,6 +5494,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 9ec7b31cce..c0e149d526 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -313,6 +313,8 @@ extern XLogRecPtr GetRedoRecPtr(void);
extern XLogRecPtr GetInsertRecPtr(void);
extern XLogRecPtr GetFlushRecPtr(void);
extern XLogRecPtr GetLastImportantRecPtr(void);
+extern void XLogArchiveWakeupStart(void);
+extern void XLogArchiveWakeupEnd(void);
extern void RemovePromoteSignalFiles(void);
extern bool PromoteIsTriggered(void);
diff --git a/src/include/access/xlogarchive.h b/src/include/access/xlogarchive.h
index 1c67de2ede..54ce0b97d7 100644
--- a/src/include/access/xlogarchive.h
+++ b/src/include/access/xlogarchive.h
@@ -25,6 +25,7 @@ extern void ExecuteRecoveryCommand(const char *command, const char *commandName,
extern void KeepFileRestoredFromArchive(const char *path, const char *xlogfname);
extern void XLogArchiveNotify(const char *xlog);
extern void XLogArchiveNotifySeg(XLogSegNo segno);
+extern void XLogArchiveWakeup(void);
extern void XLogArchiveForceDone(const char *xlog);
extern bool XLogArchiveCheckDone(const char *xlog);
extern bool XLogArchiveIsBusy(const char *xlog);
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 14fa127ab1..619b2f9c71 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index b3200874ca..e3ffc63f14 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -32,8 +32,6 @@
*/
extern int pgarch_start(void);
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern void PgArchiverMain(void) pg_attribute_noreturn();
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 56c5ec4481..c691acf8cd 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
--
2.18.2
----Next_Part(Wed_Apr__1_17_37_23_2020_570)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v30-0005-Shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v54 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 241 +++++++++--------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 160 insertions(+), 217 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..8e572afea3 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,6 +71,12 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
@@ -80,146 +84,79 @@
*/
static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
+
+ return size;
+}
+
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
+
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
+
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
+ }
+}
/*
- * pgarch_start
+ * PgArchIsSuppressed
*
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
+ * Return true if archiver relaunch is suppressed.
*
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-int
-pgarch_start(void)
+bool
+PgArchIsSuppressed(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ time_t curtime = time(NULL);
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+ if ((curtime - last_pgarch_start_time) < PGARCH_RESTART_INTERVAL)
+ return true;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
last_pgarch_start_time = curtime;
-
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
- }
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +168,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will be
+ * relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +237,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +275,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +297,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +683,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..4f8e364284 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ !PgArchIsSuppressed())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..640692d33e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchIsSuppressed(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_13_49_06_2021_682)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v53 3/7] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 227 ++++++++---------------
src/backend/postmaster/postmaster.c | 74 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 13 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 137 insertions(+), 218 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..818c9a1f3f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,70 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ Latch *latch; /* latch to wake the archiver up */
+ slock_t mutex; /* locks this struct */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
+{
+ Size size = 0;
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
+ size = add_size(size, sizeof(PgArchData));
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
-{
- time_t curtime;
- pid_t pgArchPid;
+ return size;
+}
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
+ if (!found)
{
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
-
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
-
- PgArchiverMain(0, NULL);
- break;
-#endif
-
- default:
- return (int) pgArchPid;
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ SpinLockInit(&PgArch->mutex);
+ PgArch->latch = NULL;
}
-
- /* shouldn't get here */
- return 0;
-}
-
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
-/*
- * pgarch_forkexec() -
- *
- * Format up the arglist for, then fork and exec, archive process
- */
-static pid_t
-pgarch_forkexec(void)
-{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
-
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
-
- av[ac] = NULL;
- Assert(ac < lengthof(av));
-
- return postmaster_forkexec(ac, av);
}
-#endif /* EXEC_BACKEND */
-
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +146,49 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = MyLatch;
+ SpinLockRelease(&PgArch->mutex);
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ Latch *latch;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
+ /* fetching the latch pointer might not be atomic, so use spinlock */
+ SpinLockAcquire(&PgArch->mutex);
+ latch = PgArch->latch;
+ SpinLockRelease(&PgArch->mutex);
- errno = save_errno;
+ if (latch)
+ SetLatch(latch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +213,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +251,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +273,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +659,17 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ SpinLockAcquire(&PgArch->mutex);
+ PgArch->latch = NULL;
+ SpinLockRelease(&PgArch->mutex);
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..460e629145 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -548,6 +548,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1793,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3008,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3143,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for it
+ * to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3406,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3612,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3804,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and syslogger
+ * processes are disregarded since they are not connected to shared
+ * memory; we also disregard dead_end children here. Walsenders and
+ * archiver are also disregarded, they will be terminated later after
+ * writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3911,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5037,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5134,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5188,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5429,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..d053bf1564 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,9 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_09_23_13_2021_947)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* [PATCH v55 3/3] Make archiver process an auxiliary process
@ 2021-03-11 09:01 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-11 09:01 UTC (permalink / raw)
This makes it possible to archiver process uses ipc functions and let
us monitor archiver process status.
---
doc/src/sgml/monitoring.sgml | 1 +
src/backend/access/transam/xlogarchive.c | 4 +-
src/backend/bootstrap/bootstrap.c | 22 ++-
src/backend/postmaster/pgarch.c | 240 ++++++++++-------------
src/backend/postmaster/postmaster.c | 81 ++++----
src/backend/storage/ipc/ipci.c | 2 +
src/include/miscadmin.h | 2 +
src/include/postmaster/pgarch.h | 14 +-
src/include/storage/pmsignal.h | 1 -
src/include/storage/proc.h | 8 +-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 164 insertions(+), 212 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1ba813bbb9..a96455bdb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -935,6 +935,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<literal>logical replication worker</literal>,
<literal>parallel worker</literal>, <literal>background writer</literal>,
<literal>client backend</literal>, <literal>checkpointer</literal>,
+ <literal>archiver</literal>,
<literal>startup</literal>, <literal>walreceiver</literal>,
<literal>walsender</literal> and <literal>walwriter</literal>.
In addition, background workers registered by extensions may have
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1c5a4f8b5a..26b023e754 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -25,11 +25,11 @@
#include "common/archive.h"
#include "miscadmin.h"
#include "postmaster/startup.h"
+#include "postmaster/pgarch.h"
#include "replication/walsender.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
-#include "storage/pmsignal.h"
/*
* Attempt to retrieve the specified file from off-line archival storage.
@@ -491,7 +491,7 @@ XLogArchiveNotify(const char *xlog)
/* Notify archiver that it's got something to do */
if (IsUnderPostmaster)
- SendPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER);
+ PgArchWakeup();
}
/*
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..41da0c5059 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -317,6 +317,9 @@ AuxiliaryProcessMain(int argc, char *argv[])
case StartupProcess:
MyBackendType = B_STARTUP;
break;
+ case ArchiverProcess:
+ MyBackendType = B_ARCHIVER;
+ break;
case BgWriterProcess:
MyBackendType = B_BG_WRITER;
break;
@@ -437,30 +440,29 @@ AuxiliaryProcessMain(int argc, char *argv[])
proc_exit(1); /* should never return */
case StartupProcess:
- /* don't set signals, startup process has its own agenda */
StartupProcessMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
+
+ case ArchiverProcess:
+ PgArchiverMain();
+ proc_exit(1);
case BgWriterProcess:
- /* don't set signals, bgwriter has its own agenda */
BackgroundWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case CheckpointerProcess:
- /* don't set signals, checkpointer has its own agenda */
CheckpointerMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalWriterProcess:
- /* don't set signals, walwriter has its own agenda */
InitXLOGAccess();
WalWriterMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
case WalReceiverProcess:
- /* don't set signals, walreceiver has its own agenda */
WalReceiverMain();
- proc_exit(1); /* should never return */
+ proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index edec311f12..07ba989a37 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -38,16 +38,14 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
-#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
-#include "postmaster/postmaster.h"
-#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
-#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
+#include "storage/procsignal.h"
+#include "storage/spin.h"
#include "utils/guc.h"
#include "utils/ps_status.h"
@@ -73,153 +71,101 @@
*/
#define NUM_ORPHAN_CLEANUP_RETRIES 3
+/* Shared memory area for archiver process */
+typedef struct PgArchData
+{
+ int pgprocno; /* pgprocno of archiver process */
+} PgArchData;
+
/* ----------
* Local data
* ----------
*/
-static time_t last_pgarch_start_time;
static time_t last_sigterm_time = 0;
+static PgArchData *PgArch = NULL;
/*
* Flags set by interrupt handlers for later service in the main loop.
*/
-static volatile sig_atomic_t wakened = false;
static volatile sig_atomic_t ready_to_stop = false;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgarch_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-static void pgarch_waken(SIGNAL_ARGS);
static void pgarch_waken_stop(SIGNAL_ARGS);
static void pgarch_MainLoop(void);
static void pgarch_ArchiverCopyLoop(void);
static bool pgarch_archiveXlog(char *xlog);
static bool pgarch_readyXlog(char *xlog);
static void pgarch_archiveDone(char *xlog);
+static void pgarch_die(int code, Datum arg);
-
-/* ------------------------------------------------------------
- * Public functions called from postmaster follow
- * ------------------------------------------------------------
- */
-
-/*
- * pgarch_start
- *
- * Called from postmaster at startup or after an existing archiver
- * died. Attempt to fire up a fresh archiver process.
- *
- * Returns PID of child process, or 0 if fail.
- *
- * Note: if fail, we will be called again from the postmaster main loop.
- */
-int
-pgarch_start(void)
+/* Report shared memory space needed by PgArchShmemInit */
+Size
+PgArchShmemSize(void)
{
- time_t curtime;
- pid_t pgArchPid;
+ Size size = 0;
- /*
- * Do nothing if no archiver needed
- */
- if (!XLogArchivingActive())
- return 0;
-
- /*
- * Do nothing if too soon since last archiver start. This is a safety
- * valve to protect against continuous respawn attempts if the archiver is
- * dying immediately at launch. Note that since we will be re-called from
- * the postmaster main loop, we will get another chance later.
- */
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgarch_start_time) <
- (unsigned int) PGARCH_RESTART_INTERVAL)
- return 0;
- last_pgarch_start_time = curtime;
+ size = add_size(size, sizeof(PgArchData));
-#ifdef EXEC_BACKEND
- switch ((pgArchPid = pgarch_forkexec()))
-#else
- switch ((pgArchPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork archiver: %m")));
- return 0;
-
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
-
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return size;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/* Allocate and initialize archiver-related shared memory */
+void
+PgArchShmemInit(void)
+{
+ bool found;
- PgArchiverMain(0, NULL);
- break;
-#endif
+ PgArch = (PgArchData *)
+ ShmemInitStruct("Archiver Data", PgArchShmemSize(), &found);
- default:
- return (int) pgArchPid;
+ if (!found)
+ {
+ /* First time through, so initialize */
+ MemSet(PgArch, 0, PgArchShmemSize());
+ PgArch->pgprocno = INVALID_PGPROCNO;
}
-
- /* shouldn't get here */
- return 0;
}
-/* ------------------------------------------------------------
- * Local functions called by archiver follow
- * ------------------------------------------------------------
- */
-
-
-#ifdef EXEC_BACKEND
-
/*
- * pgarch_forkexec() -
+ * PgArchIsSuppressed
*
- * Format up the arglist for, then fork and exec, archive process
+ * Return true if archiver relaunch is suppressed.
+ *
+ * This is a safety valve to protect against continuous respawn attempts if the
+ * archiver is dying immediately at launch. Note that since we will retry to
+ * launch the archiver from the postmaster main loop, we will get another
+ * chance later.
*/
-static pid_t
-pgarch_forkexec(void)
+bool
+PgArchCanRestart(void)
{
- char *av[10];
- int ac = 0;
-
- av[ac++] = "postgres";
+ static time_t last_pgarch_start_time = 0;
+ time_t curtime = time(NULL);
- av[ac++] = "--forkarch";
-
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /*
+ * allow to launch also for the first time and when something's wrong in
+ * the time values.
+ */
+ if (last_pgarch_start_time == 0 ||
+ curtime < last_pgarch_start_time ||
+ (curtime - last_pgarch_start_time) >= PGARCH_RESTART_INTERVAL)
+ {
+ last_pgarch_start_time = curtime;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ return true;
+ }
- return postmaster_forkexec(ac, av);
+ return false;
}
-#endif /* EXEC_BACKEND */
-/*
- * PgArchiverMain
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case. However,
- * since we don't use 'em, it hardly matters...
- */
-NON_EXEC_STATIC void
-PgArchiverMain(int argc, char *argv[])
+/* Main entry point for archiver process */
+void
+PgArchiverMain(void)
{
/*
* Ignore all signals usually bound to some action in the postmaster,
@@ -231,33 +177,51 @@ PgArchiverMain(int argc, char *argv[])
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, pgarch_waken);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, pgarch_waken_stop);
+
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+
+ /* Unblock signals (they were blocked when the postmaster forked us) */
PG_SETMASK(&UnBlockSig);
- MyBackendType = B_ARCHIVER;
- init_ps_display(NULL);
+ /* We shouldn't be launched unnecessarily. */
+ Assert(XLogArchivingActive());
+
+ /* Arrange to clean up at archiver exit */
+ on_shmem_exit(pgarch_die, 0);
+
+ /*
+ * Advertise our pgproc index so that backends can use our latch to wake
+ * us up while we're sleeping.
+ */
+ PgArch->pgprocno = MyProc->pgprocno;
pgarch_MainLoop();
- exit(0);
+ proc_exit(0);
}
-/* SIGUSR1 signal handler for archiver process */
-static void
-pgarch_waken(SIGNAL_ARGS)
+/*
+ * Wake up the archiver
+ */
+void
+PgArchWakeup(void)
{
- int save_errno = errno;
+ int arch_pgprocno = PgArch->pgprocno;
- /* set flag that there is work to be done */
- wakened = true;
- SetLatch(MyLatch);
-
- errno = save_errno;
+ /*
+ * We don't acquire ProcArrayLock here. It's actually fine because
+ * procLatch isn't ever freed, so we just can potentially set the wrong
+ * process' (or no process') latch. Even in that case the archiver will
+ * be relaunched shortly and will start archiving.
+ */
+ if (arch_pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
}
+
/* SIGUSR2 signal handler for archiver process */
static void
pgarch_waken_stop(SIGNAL_ARGS)
@@ -282,14 +246,6 @@ pgarch_MainLoop(void)
pg_time_t last_copy_time = 0;
bool time_to_stop;
- /*
- * We run the copy loop immediately upon entry, in case there are
- * unarchived files left over from a previous database run (or maybe the
- * archiver died unexpectedly). After that we wait for a signal or
- * timeout before doing more.
- */
- wakened = true;
-
/*
* There shouldn't be anything for the archiver to do except to wait for a
* signal ... however, the archiver exists to protect our data, so she
@@ -328,12 +284,8 @@ pgarch_MainLoop(void)
}
/* Do what we're here for */
- if (wakened || time_to_stop)
- {
- wakened = false;
- pgarch_ArchiverCopyLoop();
- last_copy_time = time(NULL);
- }
+ pgarch_ArchiverCopyLoop();
+ last_copy_time = time(NULL);
/*
* Sleep until a signal is received, or until a poll is forced by
@@ -354,13 +306,9 @@ pgarch_MainLoop(void)
WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
timeout * 1000L,
WAIT_EVENT_ARCHIVER_MAIN);
- if (rc & WL_TIMEOUT)
- wakened = true;
if (rc & WL_POSTMASTER_DEATH)
time_to_stop = true;
}
- else
- wakened = true;
}
/*
@@ -744,3 +692,15 @@ pgarch_archiveDone(char *xlog)
StatusFilePath(rlogdone, xlog, ".done");
(void) durable_rename(rlogready, rlogdone, WARNING);
}
+
+
+/*
+ * pgarch_die
+ *
+ * Exit-time cleanup handler
+ */
+static void
+pgarch_die(int code, Datum arg)
+{
+ PgArch->pgprocno = INVALID_PGPROCNO;
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index edab95a19e..e8af05c04e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -443,9 +443,10 @@ static void InitPostmasterDeathWatchHandle(void);
* even during recovery.
*/
#define PgArchStartupAllowed() \
- ((XLogArchivingActive() && pmState == PM_RUN) || \
- (XLogArchivingAlways() && \
- (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY)))
+ (((XLogArchivingActive() && pmState == PM_RUN) || \
+ (XLogArchivingAlways() && \
+ (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \
+ PgArchCanRestart())
#ifdef EXEC_BACKEND
@@ -548,6 +549,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
+#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
@@ -1792,7 +1794,7 @@ ServerLoop(void)
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/* If we need to signal the autovacuum launcher, do so now */
if (avlauncher_needs_signal)
@@ -3007,7 +3009,7 @@ reaper(SIGNAL_ARGS)
if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
if (PgStatPID == 0)
PgStatPID = pgstat_start();
@@ -3142,20 +3144,22 @@ reaper(SIGNAL_ARGS)
}
/*
- * Was it the archiver? If so, just try to start a new one; no need
- * to force reset of the rest of the system. (If fail, we'll try
- * again in future cycles of the main loop.). Unless we were waiting
- * for it to shut down; don't restart it in that case, and
+ * Was it the archiver? If exit status is zero (normal) or one (FATAL
+ * exit), we assume everything is all right just like normal backends
+ * and just try to restart a new one so that we immediately retry
+ * archiving remaining files. (If fail, we'll try again in future
+ * cycles of the postmaster's main loop.) Unless we were waiting for
+ * it to shut down; don't restart it in that case, and
* PostmasterStateMachine() will advance to the next shutdown step.
*/
if (pid == PgArchPID)
{
PgArchPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("archiver process"),
- pid, exitstatus);
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("archiver process"));
if (PgArchStartupAllowed())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
continue;
}
@@ -3403,7 +3407,7 @@ CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, or background worker.
+ * walwriter, autovacuum, archiver or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
@@ -3609,19 +3613,16 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgarch process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgArchPID != 0 && take_action)
+ /* Take care of the archiver too */
+ if (pid == PgArchPID)
+ PgArchPID = 0;
+ else if (PgArchPID != 0 && take_action)
{
ereport(DEBUG2,
(errmsg_internal("sending %s to process %d",
- "SIGQUIT",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
(int) PgArchPID)));
- signal_child(PgArchPID, SIGQUIT);
+ signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
/*
@@ -3804,12 +3805,11 @@ PostmasterStateMachine(void)
* (including autovac workers), no bgworkers (including unconnected
* ones), and no walwriter, autovac launcher or bgwriter. If we are
* doing crash recovery or an immediate shutdown then we expect the
- * checkpointer to exit as well, otherwise not. The archiver, stats,
- * and syslogger processes are disregarded since they are not
- * connected to shared memory; we also disregard dead_end children
- * here. Walsenders are also disregarded, they will be terminated
- * later after writing the checkpoint record, like the archiver
- * process.
+ * checkpointer to exit as well, otherwise not. The stats and
+ * syslogger processes are disregarded since they are not connected to
+ * shared memory; we also disregard dead_end children here. Walsenders
+ * and archiver are also disregarded, they will be terminated later
+ * after writing the checkpoint record.
*/
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
@@ -3912,6 +3912,7 @@ PostmasterStateMachine(void)
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
+ Assert(PgArchPID == 0);
/* syslogger is not considered here */
pmState = PM_NO_CHILDREN;
}
@@ -5037,12 +5038,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkarch") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgArchiverMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forkcol") == 0)
{
/* Do not want to attach to shared memory */
@@ -5140,7 +5135,7 @@ sigusr1_handler(SIGNAL_ARGS)
*/
Assert(PgArchPID == 0);
if (XLogArchivingAlways())
- PgArchPID = pgarch_start();
+ PgArchPID = StartArchiver();
/*
* If we aren't planning to enter hot standby mode later, treat
@@ -5194,16 +5189,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
- if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
- PgArchPID != 0)
- {
- /*
- * Send SIGUSR1 to archiver process, to wake it up and begin archiving
- * next WAL file.
- */
- signal_child(PgArchPID, SIGUSR1);
- }
-
/* Tell syslogger to rotate logfile if requested */
if (SysLoggerPID != 0)
{
@@ -5445,6 +5430,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork startup process: %m")));
break;
+ case ArchiverProcess:
+ ereport(LOG,
+ (errmsg("could not fork archiver process: %m")));
+ break;
case BgWriterProcess:
ereport(LOG,
(errmsg("could not fork background writer process: %m")));
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..3e4ec53a97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -144,6 +144,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, SnapMgrShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -258,6 +259,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ PgArchShmemInit();
ApplyLauncherShmemInit();
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 54693e047a..013850ac28 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -417,6 +417,7 @@ typedef enum
BootstrapProcess,
StartupProcess,
BgWriterProcess,
+ ArchiverProcess,
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -429,6 +430,7 @@ extern AuxProcType MyAuxProcType;
#define AmBootstrapProcess() (MyAuxProcType == BootstrapProcess)
#define AmStartupProcess() (MyAuxProcType == StartupProcess)
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
+#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index d102a21ab7..1e47a143e1 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -26,14 +26,10 @@
#define MAX_XFN_CHARS 40
#define VALID_XFN_CHARS "0123456789ABCDEF.history.backup.partial"
-/* ----------
- * Functions called from postmaster
- * ----------
- */
-extern int pgarch_start(void);
-
-#ifdef EXEC_BACKEND
-extern void PgArchiverMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
+extern Size PgArchShmemSize(void);
+extern void PgArchShmemInit(void);
+extern bool PgArchCanRestart(void);
+extern void PgArchiverMain(void) pg_attribute_noreturn();
+extern void PgArchWakeup(void);
#endif /* _PGARCH_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index dbbed18f61..8ed4d87ae6 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -34,7 +34,6 @@ typedef enum
{
PMSIGNAL_RECOVERY_STARTED, /* recovery has started */
PMSIGNAL_BEGIN_HOT_STANDBY, /* begin Hot Standby */
- PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */
PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */
PMSIGNAL_START_AUTOVAC_LAUNCHER, /* start an autovacuum launcher */
PMSIGNAL_START_AUTOVAC_WORKER, /* start an autovacuum worker */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index a777cb64a1..2fd1ff09a7 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -370,11 +370,11 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer and WAL writer run during normal operation.
- * Startup process and WAL receiver also consume 2 slots, but WAL writer is
- * launched only after startup has exited, so we only need 4 slots.
+ * Background writer, checkpointer, WAL writer and archiver run during normal
+ * operation. Startup process and WAL receiver also consume 2 slots, but WAL
+ * writer is launched only after startup has exited, so we only need 5 slots.
*/
-#define NUM_AUXILIARY_PROCS 4
+#define NUM_AUXILIARY_PROCS 5
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 99a34be465..78e852569e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1572,6 +1572,7 @@ PGresAttValue
PGresParamDesc
PGresult
PGresult_data
+PgArchData
PHANDLE
PLAINTREE
PLUID_AND_ATTRIBUTES
--
2.27.0
----Next_Part(Fri_Mar_12_17_24_43_2021_866)----
^ permalink raw reply [nested|flat] 279+ messages in thread
* Re: Showing applied extended statistics in explain
@ 2022-01-18 11:31 Tatsuro Yamada <[email protected]>
2022-08-01 20:38 ` Re: Showing applied extended statistics in explain Jacob Champion <[email protected]>
0 siblings, 1 reply; 279+ messages in thread
From: Tatsuro Yamada @ 2022-01-18 11:31 UTC (permalink / raw)
To: Dmitry Dolgov <[email protected]>; +Cc: Tomas Vondra <[email protected]>; pgsql-hackers
Hi Tomas!
>> What exactly use case do you have in mind?
>Well, my goal was to help users investigating the plan/estimates,
>because once you create multiple "candidate" statistics objects it may
>get quite tricky to determine which of them were applied, in what order,
>etc. It's not all that straightforward, given the various heuristics we
>use to pick the "best" statistics, apply dependencies last, etc. And I
>don't quite want to teach the users those rules, because I consider them
>mostly implementation details that wee may want to tweak in the future.
You are right. This feature will be very useful for plan tuning with
extended statistics. Therefore, I would like to help develop it. :-D
>5) It includes just statistics name + clauses, but maybe we should
>include additional info (e.g estimate for that combination of clauses).
I thought the above sentence was about what level to aim for in the initial
version. Ideally, we would like to include additional information. However,
it is clear that the more things we have, the longer it will take to
develop.
So, I think it is better to commit the initial version at a minimal level
to
provide it to users quickly.
As long as an Extended stats name is displayed in EXPLAIN result, it is
possible to figure out which extended statistics are used. That information
alone is valuable to the user.
> 4) The info is collected always, but I guess we should do that only when
>in explain mode. Not sure how expensive it is.
In the case of pg_plan_advsr that I created, I used ProcessUtility_hook to
detect EXPLAIN command. It searches all queries to find T_ExplainStmt, and
set a flag. I guess we can't use this method in Postgres core, right?
If we have to collect the extra info for all query executions, we need to
reduce overhead. I mean collecting the minimum necessary.
To do that, I think it would be better to display less Extended stats info
in EXPLAIN results. For example, displaying only extended stats names is
fine,
I guess. (I haven't understood your patch yet, so If I say wrong, sorry)
>6) The clauses in the grouping query are transformed to AND list, which
>is wrong. This is easy to fix, I was lazy to do that in a PoC patch.
6) is related 5).
If we agree to remove showing quals of extended stats in EXPLAIN result,
We can solve this problem by removing the code. Is it okay?
>2) The deparsing is modeled (i.e. copied) from how we deal with index
>quals, but it's having issues with nested OR clauses, because there are
>nested RestrictInfo nodes and the deparsing does not expect that.
>
>3) It does not work for functional dependencies, because we effectively
>"merge" all functional dependencies and apply the entries. Not sure how
>to display this, but I think it should show the individual dependencies
>actually applied.
>
>7) It does not show statistics for individual expressions. I suppose
>examine_variable could add it to the rel somehow, and maybe we could do
>that with index expressions too?
I'm not sure about 2) 3) and 7) above, so I'd like to see some concrete
examples
of queries. I would like to include it in the test pattern for regression
testing.
To be fixed:
* StatisticExtInfo should have a _copy etc. node functionality now,
right? I've hit "unrecognized node type" with a prepared statement
because it's missing.
* Is there any particular reason behind choosing only some scan nodes to
display extended stats? E.g. BitmapHeapScan is missing.
* (New) In the case of Group by, we should put Extended stats info under the
Hash/Group Aggregate node (not under Scan node).
* (New) We have to create a regression test including the above test
patterns.
Attached patch:
It is a rebased version on the head of the master because there were many
Hunks
when I applied the previous patch on master.
I'll create regression tests firstly.
Regards,
Tatsuro Yamada
Attachments:
[application/octet-stream] 0001-show-stats-in-explain-rebased.patch (16.7K, ../../CAOKkKFtc45uNFoWYOCo4St19ayxrh-_+4TnZtwxGZz6-3k_GSA@mail.gmail.com/3-0001-show-stats-in-explain-rebased.patch)
download | inline diff:
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index b970997..b351220 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -78,6 +78,9 @@ static void show_qual(List *qual, const char *qlabel,
static void show_scan_qual(List *qual, const char *qlabel,
PlanState *planstate, List *ancestors,
ExplainState *es);
+static void show_scan_stats(List *stats, List *clauses, List *ors,
+ PlanState *planstate, List *ancestors,
+ ExplainState *es);
static void show_upper_qual(List *qual, const char *qlabel,
PlanState *planstate, List *ancestors,
ExplainState *es);
@@ -1742,6 +1745,10 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (plan->qual)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
+ if (es->verbose)
+ show_scan_stats(plan->applied_stats, plan->applied_clauses,
+ plan->applied_clauses_or,
+ planstate, ancestors, es);
break;
case T_IndexOnlyScan:
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
@@ -1758,10 +1765,18 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (es->analyze)
ExplainPropertyFloat("Heap Fetches", NULL,
planstate->instrument->ntuples2, 0, es);
+ if (es->verbose)
+ show_scan_stats(plan->applied_stats, plan->applied_clauses,
+ plan->applied_clauses_or,
+ planstate, ancestors, es);
break;
case T_BitmapIndexScan:
show_scan_qual(((BitmapIndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
+ if (es->verbose)
+ show_scan_stats(plan->applied_stats, plan->applied_clauses,
+ plan->applied_clauses_or,
+ planstate, ancestors, es);
break;
case T_BitmapHeapScan:
show_scan_qual(((BitmapHeapScan *) plan)->bitmapqualorig,
@@ -1791,6 +1806,10 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (plan->qual)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
+ if (es->verbose)
+ show_scan_stats(plan->applied_stats, plan->applied_clauses,
+ plan->applied_clauses_or,
+ planstate, ancestors, es);
break;
case T_Gather:
{
@@ -1978,6 +1997,10 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (plan->qual)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
+ if (es->verbose)
+ show_scan_stats(plan->applied_stats, plan->applied_clauses,
+ plan->applied_clauses_or,
+ planstate, ancestors, es);
break;
case T_Group:
show_group_keys(castNode(GroupState, planstate), ancestors, es);
@@ -1985,6 +2008,10 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (plan->qual)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
+ if (es->verbose)
+ show_scan_stats(plan->applied_stats, plan->applied_clauses,
+ plan->applied_clauses_or,
+ planstate, ancestors, es);
break;
case T_Sort:
show_sort_keys(castNode(SortState, planstate), ancestors, es);
@@ -2305,6 +2332,76 @@ show_scan_qual(List *qual, const char *qlabel,
}
/*
+ * Show a generic expression
+ */
+static char *
+deparse_stat_expression(Node *node,
+ PlanState *planstate, List *ancestors,
+ ExplainState *es)
+{
+ List *context;
+
+ /* Set up deparsing context */
+ context = set_deparse_context_plan(es->deparse_cxt,
+ planstate->plan,
+ ancestors);
+
+ /* Deparse the expression */
+ return deparse_expression(node, context, false, false);
+}
+
+/*
+ * Show a qualifier expression (which is a List with implicit AND semantics)
+ */
+static char *
+show_stat_qual(List *qual, bool is_or,
+ PlanState *planstate, List *ancestors,
+ ExplainState *es)
+{
+ Node *node;
+
+ /* No work if empty qual */
+ if (qual == NIL)
+ return NULL;
+
+ /* Convert AND list to explicit AND */
+ if (is_or)
+ node = (Node *) make_ors_explicit(qual);
+ else
+ node = (Node *) make_ands_explicit(qual);
+
+ /* And show it */
+ return deparse_stat_expression(node, planstate, ancestors, es);
+}
+
+/*
+ * Show applied statistics for scan plan node
+ */
+static void
+show_scan_stats(List *stats, List *clauses, List *ors,
+ PlanState *planstate, List *ancestors, ExplainState *es)
+{
+ ListCell *lc1, *lc2, *lc3;
+ StringInfoData str;
+
+ forthree (lc1, stats, lc2, clauses, lc3, ors)
+ {
+ StatisticExtInfo *stat = (StatisticExtInfo *) lfirst(lc1);
+ List *clauses = (List *) lfirst(lc2);
+ int is_or = lfirst_int(lc3);
+
+ initStringInfo(&str);
+
+ appendStringInfo(&str, "%s.%s Clauses: %s",
+ get_namespace_name(get_statistics_namespace(stat->statOid)),
+ get_statistics_name(stat->statOid),
+ show_stat_qual(clauses, is_or, planstate, ancestors, es));
+
+ ExplainPropertyText("Statistics", str.data, es);
+ }
+}
+
+/*
* Show a qualifier expression for an upper-level plan node
*/
static void
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 90b5da5..654c4b5 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -136,6 +136,10 @@ CopyPlanFields(const Plan *from, Plan *newnode)
COPY_NODE_FIELD(initPlan);
COPY_BITMAPSET_FIELD(extParam);
COPY_BITMAPSET_FIELD(allParam);
+
+ COPY_NODE_FIELD(applied_stats);
+ COPY_NODE_FIELD(applied_clauses);
+ COPY_NODE_FIELD(applied_clauses_or);
}
/*
diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c
index 8223956..26b9255 100644
--- a/src/backend/nodes/makefuncs.c
+++ b/src/backend/nodes/makefuncs.c
@@ -715,6 +715,17 @@ make_ands_explicit(List *andclauses)
return make_andclause(andclauses);
}
+Expr *
+make_ors_explicit(List *orclauses)
+{
+ if (orclauses == NIL)
+ return (Expr *) makeBoolConst(true, false);
+ else if (list_length(orclauses) == 1)
+ return (Expr *) linitial(orclauses);
+ else
+ return make_orclause(orclauses);
+}
+
List *
make_ands_implicit(Expr *clause)
{
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 2b02369..6a6e20c 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -349,6 +349,10 @@ _outPlanInfo(StringInfo str, const Plan *node)
WRITE_NODE_FIELD(initPlan);
WRITE_BITMAPSET_FIELD(extParam);
WRITE_BITMAPSET_FIELD(allParam);
+
+ WRITE_NODE_FIELD(applied_stats);
+ WRITE_NODE_FIELD(applied_clauses);
+ WRITE_NODE_FIELD(applied_clauses_or);
}
/*
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3f68f7c..7220948 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1627,6 +1627,10 @@ ReadCommonPlan(Plan *local_node)
READ_NODE_FIELD(initPlan);
READ_BITMAPSET_FIELD(extParam);
READ_BITMAPSET_FIELD(allParam);
+
+ READ_NODE_FIELD(applied_stats);
+ READ_NODE_FIELD(applied_clauses);
+ READ_NODE_FIELD(applied_clauses_or);
}
/*
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index cd6d72c..9efb961 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -5294,12 +5294,27 @@ order_qual_clauses(PlannerInfo *root, List *clauses)
static void
copy_generic_path_info(Plan *dest, Path *src)
{
+ ListCell *lc;
+
dest->startup_cost = src->startup_cost;
dest->total_cost = src->total_cost;
dest->plan_rows = src->rows;
dest->plan_width = src->pathtarget->width;
dest->parallel_aware = src->parallel_aware;
dest->parallel_safe = src->parallel_safe;
+
+ dest->applied_stats = src->parent->applied_stats;
+ dest->applied_clauses_or = src->parent->applied_clauses_or;
+
+ dest->applied_clauses = NIL;
+ foreach (lc, src->parent->applied_clauses)
+ {
+ List *clauses = (List *) lfirst(lc);
+
+ dest->applied_clauses
+ = lappend(dest->applied_clauses,
+ maybe_extract_actual_clauses(clauses, false));
+ }
}
/*
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index 520409f..6a4e1c1 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -260,6 +260,10 @@ build_simple_rel(PlannerInfo *root, int relid, RelOptInfo *parent)
rel->partexprs = NULL;
rel->nullable_partexprs = NULL;
+ rel->applied_stats = NIL;
+ rel->applied_clauses = NIL;
+ rel->applied_clauses_or = NIL;
+
/*
* Pass assorted information down the inheritance hierarchy.
*/
@@ -675,6 +679,10 @@ build_join_rel(PlannerInfo *root,
joinrel->partexprs = NULL;
joinrel->nullable_partexprs = NULL;
+ joinrel->applied_stats = NIL;
+ joinrel->applied_clauses = NIL;
+ joinrel->applied_clauses_or = NIL;
+
/* Compute information relevant to the foreign relations. */
set_foreign_rel_properties(joinrel, outer_rel, inner_rel);
@@ -854,6 +862,10 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel,
joinrel->partexprs = NULL;
joinrel->nullable_partexprs = NULL;
+ joinrel->applied_stats = NIL;
+ joinrel->applied_clauses = NIL;
+ joinrel->applied_clauses_or = NIL;
+
joinrel->top_parent_relids = bms_union(outer_rel->top_parent_relids,
inner_rel->top_parent_relids);
diff --git a/src/backend/optimizer/util/restrictinfo.c b/src/backend/optimizer/util/restrictinfo.c
index ef8df3d..cd982c3 100644
--- a/src/backend/optimizer/util/restrictinfo.c
+++ b/src/backend/optimizer/util/restrictinfo.c
@@ -459,6 +459,41 @@ extract_actual_clauses(List *restrictinfo_list,
}
/*
+ * maybe_extract_actual_clauses
+ *
+ * Just like extract_actual_clauses, but does not require the clauses to
+ * already be RestrictInfo.
+ *
+ * XXX Does not handle RestrictInfos nested in OR clauses.
+ */
+List *
+maybe_extract_actual_clauses(List *restrictinfo_list,
+ bool pseudoconstant)
+{
+ List *result = NIL;
+ ListCell *l;
+
+ foreach(l, restrictinfo_list)
+ {
+ RestrictInfo *rinfo;
+ Node *node = (Node *) lfirst(l);
+
+ if (!IsA(node, RestrictInfo))
+ {
+ result = lappend(result, node);
+ continue;
+ }
+
+ rinfo = (RestrictInfo *) node;
+
+ if (rinfo->pseudoconstant == pseudoconstant)
+ result = lappend(result, rinfo->clause);
+ }
+
+ return result;
+}
+
+/*
* extract_actual_join_clauses
*
* Extract bare clauses from 'restrictinfo_list', separating those that
diff --git a/src/backend/statistics/extended_stats.c b/src/backend/statistics/extended_stats.c
index 87fe82e..3606e35 100644
--- a/src/backend/statistics/extended_stats.c
+++ b/src/backend/statistics/extended_stats.c
@@ -1829,6 +1829,11 @@ statext_mcv_clauselist_selectivity(PlannerInfo *root, List *clauses, int varReli
list_exprs[listidx] = NULL;
}
+ /* add it to the list of applied stats/clauses */
+ rel->applied_stats = lappend(rel->applied_stats, stat);
+ rel->applied_clauses = lappend(rel->applied_clauses, stat_clauses);
+ rel->applied_clauses_or = lappend_int(rel->applied_clauses_or, (is_or) ? 1 : 0);
+
if (is_or)
{
bool *or_matches = NULL;
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index 1fbb0b2..c03a2e3 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -4019,6 +4019,7 @@ estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel,
ListCell *lc2;
Bitmapset *matched = NULL;
AttrNumber attnum_offset;
+ List *matched_exprs = NIL;
/*
* How much we need to offset the attnums? If there are no
@@ -4066,6 +4067,9 @@ estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel,
matched = bms_add_member(matched, attnum);
+ /* track expressions matched by this statistics */
+ matched_exprs = lappend(matched_exprs, varinfo->var);
+
found = true;
}
@@ -4094,6 +4098,9 @@ estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel,
matched = bms_add_member(matched, attnum);
+ /* track expressions matched by this statistics */
+ matched_exprs = lappend(matched_exprs, expr);
+
/* there should be just one matching expression */
break;
}
@@ -4102,6 +4109,10 @@ estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel,
}
}
+ rel->applied_stats = lappend(rel->applied_stats, matched_info);
+ rel->applied_clauses = lappend(rel->applied_clauses, matched_exprs);
+ rel->applied_clauses_or = lappend(rel->applied_clauses_or, 0);
+
/* Find the specific item that exactly matches the combination */
for (i = 0; i < stats->nitems; i++)
{
diff --git a/src/backend/utils/cache/lsyscache.c b/src/backend/utils/cache/lsyscache.c
index feef999..3363be4 100644
--- a/src/backend/utils/cache/lsyscache.c
+++ b/src/backend/utils/cache/lsyscache.c
@@ -33,6 +33,7 @@
#include "catalog/pg_proc.h"
#include "catalog/pg_range.h"
#include "catalog/pg_statistic.h"
+#include "catalog/pg_statistic_ext.h"
#include "catalog/pg_transform.h"
#include "catalog/pg_type.h"
#include "miscadmin.h"
@@ -3568,3 +3569,51 @@ get_index_isclustered(Oid index_oid)
return isclustered;
}
+
+/*
+ * get_statistics_name
+ * Returns the name of a given extended statistics
+ *
+ * Returns a palloc'd copy of the string, or NULL if no such namespace.
+ */
+char *
+get_statistics_name(Oid stxid)
+{
+ HeapTuple tp;
+
+ tp = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(stxid));
+ if (HeapTupleIsValid(tp))
+ {
+ Form_pg_statistic_ext stxtup = (Form_pg_statistic_ext) GETSTRUCT(tp);
+ char *result;
+
+ result = pstrdup(NameStr(stxtup->stxname));
+ ReleaseSysCache(tp);
+ return result;
+ }
+ else
+ return NULL;
+}
+
+/*
+ * get_statistics_namespace
+ * Returns the namespace OID of a given extended statistics
+ */
+Oid
+get_statistics_namespace(Oid stxid)
+{
+ HeapTuple tp;
+
+ tp = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(stxid));
+ if (HeapTupleIsValid(tp))
+ {
+ Form_pg_statistic_ext stxtup = (Form_pg_statistic_ext) GETSTRUCT(tp);
+ Oid result;
+
+ result = stxtup->stxnamespace;
+ ReleaseSysCache(tp);
+ return result;
+ }
+ else
+ return InvalidOid;
+}
diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h
index fe17310..249833d 100644
--- a/src/include/nodes/makefuncs.h
+++ b/src/include/nodes/makefuncs.h
@@ -94,6 +94,8 @@ extern Node *make_and_qual(Node *qual1, Node *qual2);
extern Expr *make_ands_explicit(List *andclauses);
extern List *make_ands_implicit(Expr *clause);
+extern Expr *make_ors_explicit(List *orclauses);
+
extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid,
List *expressions, List *predicates,
bool unique, bool isready, bool concurrent);
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 1f3845b..6abfea6 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -773,6 +773,11 @@ typedef struct RelOptInfo
Relids all_partrels; /* Relids set of all partition relids */
List **partexprs; /* Non-nullable partition key expressions */
List **nullable_partexprs; /* Nullable partition key expressions */
+
+ /* info about applied extended statistics */
+ List *applied_stats; /* list of StatisticExtInfo */
+ List *applied_clauses; /* list of lists (of clauses) */
+ List *applied_clauses_or; /* are the clauses AND or OR */
} RelOptInfo;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 0b518ce..425d659 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -158,6 +158,11 @@ typedef struct Plan
*/
Bitmapset *extParam;
Bitmapset *allParam;
+
+ /* info about applied statistics */
+ List *applied_stats;
+ List *applied_clauses;
+ List *applied_clauses_or;
} Plan;
/* ----------------
diff --git a/src/include/optimizer/restrictinfo.h b/src/include/optimizer/restrictinfo.h
index 6d30bd5..fc27723 100644
--- a/src/include/optimizer/restrictinfo.h
+++ b/src/include/optimizer/restrictinfo.h
@@ -37,6 +37,8 @@ extern bool restriction_is_securely_promotable(RestrictInfo *restrictinfo,
extern List *get_actual_clauses(List *restrictinfo_list);
extern List *extract_actual_clauses(List *restrictinfo_list,
bool pseudoconstant);
+extern List *maybe_extract_actual_clauses(List *restrictinfo_list,
+ bool pseudoconstant);
extern void extract_actual_join_clauses(List *restrictinfo_list,
Relids joinrelids,
List **joinquals,
diff --git a/src/include/utils/lsyscache.h b/src/include/utils/lsyscache.h
index b8dd27d..a9c206a 100644
--- a/src/include/utils/lsyscache.h
+++ b/src/include/utils/lsyscache.h
@@ -199,6 +199,9 @@ extern bool get_index_isreplident(Oid index_oid);
extern bool get_index_isvalid(Oid index_oid);
extern bool get_index_isclustered(Oid index_oid);
+extern char *get_statistics_name(Oid stxid);
+extern Oid get_statistics_namespace(Oid stxid);
+
#define type_is_array(typid) (get_element_type(typid) != InvalidOid)
/* type_is_array_domain accepts both plain arrays and domains over arrays */
#define type_is_array_domain(typid) (get_base_element_type(typid) != InvalidOid)
^ permalink raw reply [nested|flat] 279+ messages in thread
* Re: Showing applied extended statistics in explain
2022-01-18 11:31 Re: Showing applied extended statistics in explain Tatsuro Yamada <[email protected]>
@ 2022-08-01 20:38 ` Jacob Champion <[email protected]>
0 siblings, 0 replies; 279+ messages in thread
From: Jacob Champion @ 2022-08-01 20:38 UTC (permalink / raw)
To: Tatsuro Yamada <[email protected]>; Dmitry Dolgov <[email protected]>; +Cc: Tomas Vondra <[email protected]>; pgsql-hackers
As discussed in [1], we're taking this opportunity to return some
patchsets that don't appear to be getting enough reviewer interest.
This is not a rejection, since we don't necessarily think there's
anything unacceptable about the entry, but it differs from a standard
"Returned with Feedback" in that there's probably not much actionable
feedback at all. Rather than code changes, what this patch needs is more
community interest. You might
- ask people for help with your approach,
- see if there are similar patches that your code could supplement,
- get interested parties to agree to review your patch in a CF, or
- possibly present the functionality in a way that's easier to review
overall.
(Doing these things is no guarantee that there will be interest, but
it's hopefully better than endlessly rebasing a patchset that is not
receiving any feedback from the community.)
Once you think you've built up some community support and the patchset
is ready for review, you (or any interested party) can resurrect the
patch entry by visiting
https://commitfest.postgresql.org/38/3050/
and changing the status to "Needs Review", and then changing the
status again to "Move to next CF". (Don't forget the second step;
hopefully we will have streamlined this in the near future!)
Thanks,
--Jacob
[1] https://postgr.es/m/[email protected]
^ permalink raw reply [nested|flat] 279+ messages in thread
end of thread, other threads:[~2022-08-01 20:38 UTC | newest]
Thread overview: 279+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-03-01 22:02 Re: Disallowing multiple queries per PQexec() Jim Nasby <[email protected]>
2017-03-02 09:01 ` Surafel Temesgen <[email protected]>
2018-11-07 07:53 [PATCH 3/5] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2018-11-07 07:53 [PATCH v24 3/5] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v32 4/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v44 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v37 3/6] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v42 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v41 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v43 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v33 4/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v29 4/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v36 4/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v25 4/8] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v35 4/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v31 4/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v40 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v39 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v38 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v34 4/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v49 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v52 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v33 4/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v46 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v45 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v51 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v48 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v47 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v50 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2020-03-13 07:59 [PATCH v30 4/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v54 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v53 3/7] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2021-03-11 09:01 [PATCH v55 3/3] Make archiver process an auxiliary process Kyotaro Horiguchi <[email protected]>
2022-01-18 11:31 Re: Showing applied extended statistics in explain Tatsuro Yamada <[email protected]>
2022-08-01 20:38 ` Re: Showing applied extended statistics in explain Jacob Champion <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox