From: Bossart, Nathan <bossartn@amazon.com>
To: David Steele <david@pgmasters.net>
To: Fujii Masao <masao.fujii@oss.nttdata.com>
To: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: Magnus Hagander <magnus@hagander.net>
Cc: Stephen Frost <sfrost@snowman.net>
Subject: Re: archive modules
Date: Wed, 5 Jan 2022 23:10:50 +0000
Message-ID: <3842CF04-528B-42A5-A12D-BA3970EF1E53@amazon.com> (raw)
In-Reply-To: <90D3CEE4-00C1-450C-80B1-88B69D9318DE@amazon.com>
References: <668D2428-F73B-475E-87AE-F89D67942270@amazon.com>
<e437bf15-9917-7b78-f326-90b9c57ba8ae@oss.nttdata.com>
<F9F7CCA3-489D-479E-941F-84C5C1D1D3CB@amazon.com>
<db01f425-2689-276d-f81a-e8cd560816b2@oss.nttdata.com>
<53a180a7-bf1b-38c5-7fb4-4088dc022b98@pgmasters.net>
<A30D8D33-8944-4898-BCA8-C77C18258247@amazon.com>
<4c01ce36-fbed-104a-ef9c-6046343dc845@pgmasters.net>
<316CAF77-A0AC-49AE-A175-1C1ECF6C1981@amazon.com>
<71E7A5D1-E418-4516-A21D-BD838BA759CE@amazon.com>
<0CF39706-6952-49B1-987D-F012CDF5EEC4@amazon.com>
<90D3CEE4-00C1-450C-80B1-88B69D9318DE@amazon.com>
On 11/22/21, 10:01 AM, "Bossart, Nathan" <bossartn@amazon.com> wrote:
> On 11/19/21, 11:24 AM, "Bossart, Nathan" <bossartn@amazon.com> wrote:
>> I went ahead and split the patch into 4 separate patches in an effort
>> to ease review. 0001 just refactors the shell archiving logic to its
>> own file. 0002 introduces the archive modules infrastructure. 0003
>> introduces the basic_archive test module. And 0004 is the docs.
>
> Here is a rebased patch set (1b06d7b broke v10).
I'm attempting to make cfbot happy again with v12. It looked like
there was a missing #include for Windows.
Nathan
Attachments:
[application/octet-stream] v12-0001-Refactor-shell-archive-function-to-its-own-file.patch (8.9K, ../3842CF04-528B-42A5-A12D-BA3970EF1E53@amazon.com/2-v12-0001-Refactor-shell-archive-function-to-its-own-file.patch)
download | inline diff:
From e44a714575df136409951ffa55cb5f75f1ba6cbd Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Mon, 22 Nov 2021 17:49:18 +0000
Subject: [PATCH v12 1/4] Refactor shell archive function to its own file.
---
src/backend/postmaster/Makefile | 1 +
src/backend/postmaster/pgarch.c | 125 ++----------------------------
src/backend/postmaster/shell_archive.c | 135 +++++++++++++++++++++++++++++++++
src/include/postmaster/pgarch.h | 3 +
4 files changed, 145 insertions(+), 119 deletions(-)
create mode 100644 src/backend/postmaster/shell_archive.c
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefileindex 787c6a2c3b..dbbeac5a82 100644--- a/src/backend/postmaster/Makefile+++ b/src/backend/postmaster/Makefile@@ -23,6 +23,7 @@ OBJS = \
pgarch.o \
pgstat.o \
postmaster.o \
+ shell_archive.o \
startup.o \
syslogger.o \
walwriter.o
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.cindex 5b6bf9f4e0..dc68dd570e 100644--- a/src/backend/postmaster/pgarch.c+++ b/src/backend/postmaster/pgarch.c@@ -25,19 +25,14 @@
*/
#include "postgres.h"
-#include <fcntl.h>-#include <signal.h>
#include <time.h>
#include <sys/stat.h>
-#include <sys/time.h>-#include <sys/wait.h>
#include <unistd.h>
#include "access/xlog.h"
#include "access/xlog_internal.h"
#include "lib/binaryheap.h"
#include "libpq/pqsignal.h"
-#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
@@ -503,132 +498,24 @@ pgarch_ArchiverCopyLoop(void)
static bool
pgarch_archiveXlog(char *xlog)
{
- char xlogarchcmd[MAXPGPATH];
char pathname[MAXPGPATH];
char activitymsg[MAXFNAMELEN + 16];
- char *dp;- char *endp;- const char *sp;- int rc;+ bool ret;
snprintf(pathname, MAXPGPATH, XLOGDIR "/%s", xlog);
- /*- * construct the command to be executed- */- dp = xlogarchcmd;- endp = xlogarchcmd + MAXPGPATH - 1;- *endp = '\0';-- for (sp = XLogArchiveCommand; *sp; sp++)- {- if (*sp == '%')- {- switch (sp[1])- {- case 'p':- /* %p: relative path of source file */- sp++;- strlcpy(dp, pathname, endp - dp);- make_native_path(dp);- dp += strlen(dp);- break;- case 'f':- /* %f: filename of source file */- sp++;- strlcpy(dp, xlog, endp - dp);- dp += strlen(dp);- break;- case '%':- /* convert %% to a single % */- sp++;- if (dp < endp)- *dp++ = *sp;- break;- default:- /* otherwise treat the % as not special */- if (dp < endp)- *dp++ = *sp;- break;- }- }- else- {- if (dp < endp)- *dp++ = *sp;- }- }- *dp = '\0';-- ereport(DEBUG3,- (errmsg_internal("executing archive command \"%s\"",- xlogarchcmd)));-
/* Report archive activity in PS display */
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- pgstat_report_wait_start(WAIT_EVENT_ARCHIVE_COMMAND);- rc = system(xlogarchcmd);- pgstat_report_wait_end();-- if (rc != 0)- {- /*- * If either the shell itself, or a called command, died on a signal,- * abort the archiver. We do this because system() ignores SIGINT and- * SIGQUIT while waiting; so a signal is very likely something that- * should have interrupted us too. Also die if the shell got a hard- * "command not found" type of error. If we overreact it's no big- * deal, the postmaster will just start the archiver again.- */- int lev = wait_result_is_any_signal(rc, true) ? FATAL : LOG;-- if (WIFEXITED(rc))- {- ereport(lev,- (errmsg("archive command failed with exit code %d",- WEXITSTATUS(rc)),- errdetail("The failed archive command was: %s",- xlogarchcmd)));- }- else if (WIFSIGNALED(rc))- {-#if defined(WIN32)- ereport(lev,- (errmsg("archive command was terminated by exception 0x%X",- WTERMSIG(rc)),- errhint("See C include file \"ntstatus.h\" for a description of the hexadecimal value."),- errdetail("The failed archive command was: %s",- xlogarchcmd)));-#else- ereport(lev,- (errmsg("archive command was terminated by signal %d: %s",- WTERMSIG(rc), pg_strsignal(WTERMSIG(rc))),- errdetail("The failed archive command was: %s",- xlogarchcmd)));-#endif- }- else- {- ereport(lev,- (errmsg("archive command exited with unrecognized status %d",- rc),- errdetail("The failed archive command was: %s",- xlogarchcmd)));- }-+ ret = shell_archive_file(xlog, pathname);+ if (ret)+ snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);+ else
snprintf(activitymsg, sizeof(activitymsg), "failed on %s", xlog);
- set_ps_display(activitymsg);-- return false;- }- elog(DEBUG1, "archived write-ahead log file \"%s\"", xlog);-- snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
set_ps_display(activitymsg);
- return true;+ return ret;
}
/*
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/postmaster/shell_archive.cnew file mode 100644index 0000000000..b54e701da4--- /dev/null+++ b/src/backend/postmaster/shell_archive.c@@ -0,0 +1,135 @@+/*-------------------------------------------------------------------------+ *+ * shell_archive.c+ *+ * Copyright (c) 2022, PostgreSQL Global Development Group+ *+ * IDENTIFICATION+ * src/backend/postmaster/shell_archive.c+ *+ *-------------------------------------------------------------------------+ */+#include "postgres.h"++#include <sys/wait.h>++#include "access/xlog.h"+#include "pgstat.h"+#include "postmaster/pgarch.h"++bool+shell_archive_file(const char *file, const char *path)+{+ char xlogarchcmd[MAXPGPATH];+ char *dp;+ char *endp;+ const char *sp;+ int rc;++ /*+ * construct the command to be executed+ */+ dp = xlogarchcmd;+ endp = xlogarchcmd + MAXPGPATH - 1;+ *endp = '\0';++ for (sp = XLogArchiveCommand; *sp; sp++)+ {+ if (*sp == '%')+ {+ switch (sp[1])+ {+ case 'p':+ /* %p: relative path of source file */+ sp++;+ strlcpy(dp, path, endp - dp);+ make_native_path(dp);+ dp += strlen(dp);+ break;+ case 'f':+ /* %f: filename of source file */+ sp++;+ strlcpy(dp, file, endp - dp);+ dp += strlen(dp);+ break;+ case '%':+ /* convert %% to a single % */+ sp++;+ if (dp < endp)+ *dp++ = *sp;+ break;+ default:+ /* otherwise treat the % as not special */+ if (dp < endp)+ *dp++ = *sp;+ break;+ }+ }+ else+ {+ if (dp < endp)+ *dp++ = *sp;+ }+ }+ *dp = '\0';++ ereport(DEBUG3,+ (errmsg_internal("executing archive command \"%s\"",+ xlogarchcmd)));++ pgstat_report_wait_start(WAIT_EVENT_ARCHIVE_COMMAND);+ rc = system(xlogarchcmd);+ pgstat_report_wait_end();++ if (rc != 0)+ {+ /*+ * If either the shell itself, or a called command, died on a signal,+ * abort the archiver. We do this because system() ignores SIGINT and+ * SIGQUIT while waiting; so a signal is very likely something that+ * should have interrupted us too. Also die if the shell got a hard+ * "command not found" type of error. If we overreact it's no big+ * deal, the postmaster will just start the archiver again.+ */+ int lev = wait_result_is_any_signal(rc, true) ? FATAL : LOG;++ if (WIFEXITED(rc))+ {+ ereport(lev,+ (errmsg("archive command failed with exit code %d",+ WEXITSTATUS(rc)),+ errdetail("The failed archive command was: %s",+ xlogarchcmd)));+ }+ else if (WIFSIGNALED(rc))+ {+#if defined(WIN32)+ ereport(lev,+ (errmsg("archive command was terminated by exception 0x%X",+ WTERMSIG(rc)),+ errhint("See C include file \"ntstatus.h\" for a description of the hexadecimal value."),+ errdetail("The failed archive command was: %s",+ xlogarchcmd)));+#else+ ereport(lev,+ (errmsg("archive command was terminated by signal %d: %s",+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc))),+ errdetail("The failed archive command was: %s",+ xlogarchcmd)));+#endif+ }+ else+ {+ ereport(lev,+ (errmsg("archive command exited with unrecognized status %d",+ rc),+ errdetail("The failed archive command was: %s",+ xlogarchcmd)));+ }++ return false;+ }++ elog(DEBUG1, "archived write-ahead log file \"%s\"", file);+ return true;+}diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.hindex 732615be57..9c4bd69b56 100644--- a/src/include/postmaster/pgarch.h+++ b/src/include/postmaster/pgarch.h@@ -33,4 +33,7 @@ extern void PgArchiverMain(void) pg_attribute_noreturn();
extern void PgArchWakeup(void);
extern void PgArchForceDirScan(void);
+/* in shell_archive.c */+extern bool shell_archive_file(const char *file, const char *path);+
#endif /* _PGARCH_H */
--
2.16.6
[application/octet-stream] v12-0002-Introduce-archive-modules-infrastructure.patch (10.7K, ../3842CF04-528B-42A5-A12D-BA3970EF1E53@amazon.com/3-v12-0002-Introduce-archive-modules-infrastructure.patch)
download | inline diff:
From 76e0d1b1c32cba4d1b4cfe9f3d71c9d43a3e9b6c Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Fri, 19 Nov 2021 01:04:41 +0000
Subject: [PATCH v12 2/4] Introduce archive modules infrastructure.
---
src/backend/access/transam/xlog.c | 2 +-
src/backend/postmaster/pgarch.c | 71 +++++++++++++++++++++++++--
src/backend/postmaster/shell_archive.c | 24 ++++++++-
src/backend/utils/init/miscinit.c | 1 +
src/backend/utils/misc/guc.c | 12 ++++-
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/xlog.h | 1 -
src/include/postmaster/pgarch.h | 46 ++++++++++++++++-
8 files changed, 147 insertions(+), 11 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.cindex 87cd05c945..fa13bf4d21 100644--- a/src/backend/access/transam/xlog.c+++ b/src/backend/access/transam/xlog.c@@ -8832,7 +8832,7 @@ ShutdownXLOG(int code, Datum arg)
* process one more time at the end of shutdown). The checkpoint
* record will go to the next XLOG file and won't be archived (yet).
*/
- if (XLogArchivingActive() && XLogArchiveCommandSet())+ if (XLogArchivingActive())
RequestXLogSwitch(false);
CreateCheckPoint(CHECKPOINT_IS_SHUTDOWN | CHECKPOINT_IMMEDIATE);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.cindex dc68dd570e..61e88ff133 100644--- a/src/backend/postmaster/pgarch.c+++ b/src/backend/postmaster/pgarch.c@@ -88,6 +88,8 @@ typedef struct PgArchData
slock_t arch_lck;
} PgArchData;
+char *XLogArchiveLibrary = "";+
/* ----------
* Local data
@@ -95,6 +97,8 @@ typedef struct PgArchData
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
+static ArchiveModuleCallbacks *ArchiveContext = NULL;+
/*
* Stuff for tracking multiple files to archive from each scan of
@@ -139,6 +143,7 @@ static void pgarch_archiveDone(char *xlog);
static void pgarch_die(int code, Datum arg);
static void HandlePgArchInterrupts(void);
static int ready_file_comparator(Datum a, Datum b, void *arg);
+static void LoadArchiveLibrary(void);
/* Report shared memory space needed by PgArchShmemInit */
Size
@@ -235,6 +240,11 @@ PgArchiverMain(void)
*/
PgArch->pgprocno = MyProc->pgprocno;
+ /*+ * Load the archive_library.+ */+ LoadArchiveLibrary();+
/* Create workspace for pgarch_readyXlog() */
arch_files = palloc(sizeof(struct arch_files_state));
arch_files->arch_files_size = 0;
@@ -406,11 +416,11 @@ pgarch_ArchiverCopyLoop(void)
*/
HandlePgArchInterrupts();
- /* can't do anything if no command ... */- if (!XLogArchiveCommandSet())+ /* can't do anything if not configured ... */+ if (!ArchiveContext->check_configured_cb())
{
ereport(WARNING,
- (errmsg("archive_mode enabled, yet archive_command is not set")));+ (errmsg("archive_mode enabled, yet archiving is not configured")));
return;
}
@@ -491,7 +501,7 @@ pgarch_ArchiverCopyLoop(void)
/*
* pgarch_archiveXlog
*
- * Invokes system(3) to copy one archive file to wherever it should go+ * Invokes archive_file_cb to copy one archive file to wherever it should go
*
* Returns true if successful
*/
@@ -508,7 +518,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = shell_archive_file(xlog, pathname);+ ret = ArchiveContext->archive_file_cb(xlog, pathname);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -761,5 +771,56 @@ HandlePgArchInterrupts(void)
{
ConfigReloadPending = false;
ProcessConfigFile(PGC_SIGHUP);
++ /*+ * Load the archive_library in case it changed. Ideally, this would+ * first unload any pre-existing loaded archive library to release+ * custom GUCs, decommission background workers, etc., but there is+ * presently no mechanism for unloading a library. For more+ * information, see the comment above internal_unload_library().+ */+ LoadArchiveLibrary();
}
}
++/*+ * LoadArchiveLibrary+ *+ * Loads the archiving callbacks into our local ArchiveContext.+ */+static void+LoadArchiveLibrary(void)+{+ ArchiveModuleInit archive_init;++ if (ArchiveContext == NULL)+ ArchiveContext = palloc(sizeof(ArchiveModuleCallbacks));++ memset(ArchiveContext, 0, sizeof(ArchiveModuleCallbacks));++ /*+ * If shell archiving is enabled, use our special initialization+ * function. Otherwise, load the library and call its+ * _PG_archive_module_init().+ */+ if (ShellArchivingEnabled())+ archive_init = shell_archive_init;+ else+ archive_init = (ArchiveModuleInit)+ load_external_function(XLogArchiveLibrary,+ "_PG_archive_module_init", false, NULL);++ if (archive_init == NULL)+ ereport(ERROR,+ (errmsg("archive modules have to declare the "+ "_PG_archive_module_init symbol")));++ archive_init(ArchiveContext);++ if (ArchiveContext->check_configured_cb == NULL)+ ereport(ERROR,+ (errmsg("archive modules must register a check callback")));+ if (ArchiveContext->archive_file_cb == NULL)+ ereport(ERROR,+ (errmsg("archive modules must register an archive callback")));+}diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/postmaster/shell_archive.cindex b54e701da4..19e240c205 100644--- a/src/backend/postmaster/shell_archive.c+++ b/src/backend/postmaster/shell_archive.c@@ -2,6 +2,10 @@
*
* shell_archive.c
*
+ * This archiving function uses a user-specified shell command (the+ * archive_command GUC) to copy write-ahead log files. It is used as the+ * default, but other modules may define their own custom archiving logic.+ *
* Copyright (c) 2022, PostgreSQL Global Development Group
*
* IDENTIFICATION
@@ -17,7 +21,25 @@
#include "pgstat.h"
#include "postmaster/pgarch.h"
-bool+static bool shell_archive_configured(void);+static bool shell_archive_file(const char *file, const char *path);++void+shell_archive_init(ArchiveModuleCallbacks *cb)+{+ AssertVariableIsOfType(&shell_archive_init, ArchiveModuleInit);++ cb->check_configured_cb = shell_archive_configured;+ cb->archive_file_cb = shell_archive_file;+}++static bool+shell_archive_configured(void)+{+ return XLogArchiveCommand[0] != '\0';+}++static bool
shell_archive_file(const char *file, const char *path)
{
char xlogarchcmd[MAXPGPATH];
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.cindex 88801374b5..358d9ed029 100644--- a/src/backend/utils/init/miscinit.c+++ b/src/backend/utils/init/miscinit.c@@ -38,6 +38,7 @@
#include "pgstat.h"
#include "postmaster/autovacuum.h"
#include "postmaster/interrupt.h"
+#include "postmaster/pgarch.h"
#include "postmaster/postmaster.h"
#include "storage/fd.h"
#include "storage/ipc.h"
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.cindex f9504d3aec..2554923e1d 100644--- a/src/backend/utils/misc/guc.c+++ b/src/backend/utils/misc/guc.c@@ -3869,13 +3869,23 @@ static struct config_string ConfigureNamesString[] =
{
{"archive_command", PGC_SIGHUP, WAL_ARCHIVING,
gettext_noop("Sets the shell command that will be called to archive a WAL file."),
- NULL+ gettext_noop("This is unused if \"archive_library\" does not indicate archiving via shell is enabled.")
},
&XLogArchiveCommand,
"",
NULL, NULL, show_archive_command
},
+ {+ {"archive_library", PGC_SIGHUP, WAL_ARCHIVING,+ gettext_noop("Sets the library that will be called to archive a WAL file."),+ gettext_noop("A value of \"shell\" or an empty string indicates that \"archive_command\" should be used.")+ },+ &XLogArchiveLibrary,+ "shell",+ NULL, NULL, NULL+ },+
{
{"restore_command", PGC_SIGHUP, WAL_ARCHIVE_RECOVERY,
gettext_noop("Sets the shell command that will be called to retrieve an archived WAL file."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sampleindex a1acd46b61..e8bdd1fe13 100644--- a/src/backend/utils/misc/postgresql.conf.sample+++ b/src/backend/utils/misc/postgresql.conf.sample@@ -245,6 +245,7 @@
#archive_mode = off # enables archiving; off, on, or always
# (change requires restart)
+#archive_library = 'shell' # library to use to archive a logfile segment
#archive_command = '' # command to use to archive a logfile segment
# placeholders: %p = path of file to archive
# %f = file name only
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.hindex 34f6c89f06..b7dfb580ad 100644--- a/src/include/access/xlog.h+++ b/src/include/access/xlog.h@@ -155,7 +155,6 @@ extern PGDLLIMPORT int wal_level;
/* Is WAL archiving enabled always (even during recovery)? */
#define XLogArchivingAlways() \
(AssertMacro(XLogArchiveMode == ARCHIVE_MODE_OFF || wal_level >= WAL_LEVEL_REPLICA), XLogArchiveMode == ARCHIVE_MODE_ALWAYS)
-#define XLogArchiveCommandSet() (XLogArchiveCommand[0] != '\0')
/*
* Is WAL-logging necessary for archival or log-shipping, or can we skip
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.hindex 9c4bd69b56..03b5ab7c22 100644--- a/src/include/postmaster/pgarch.h+++ b/src/include/postmaster/pgarch.h@@ -33,7 +33,49 @@ extern void PgArchiverMain(void) pg_attribute_noreturn();
extern void PgArchWakeup(void);
extern void PgArchForceDirScan(void);
-/* in shell_archive.c */-extern bool shell_archive_file(const char *file, const char *path);+/*+ * The value of the archive_library GUC.+ */+extern char *XLogArchiveLibrary;++/*+ * Callback that gets called to determine if the archive module is+ * configured.+ */+typedef bool (*ArchiveCheckConfiguredCB) (void);++/*+ * Callback called to archive a single WAL file.+ */+typedef bool (*ArchiveFileCB) (const char *file, const char *path);++/*+ * Archive module callbacks+ */+typedef struct ArchiveModuleCallbacks+{+ ArchiveCheckConfiguredCB check_configured_cb;+ ArchiveFileCB archive_file_cb;+} ArchiveModuleCallbacks;++/*+ * Type of the shared library symbol _PG_archive_module_init that is looked+ * up when loading an archive library.+ */+typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);++/*+ * Since the logic for archiving via a shell command is in the core server+ * and does not need to be loaded via a shared library, it has a special+ * initialization function.+ */+extern void shell_archive_init(ArchiveModuleCallbacks *cb);++/*+ * We consider archiving via shell to be enabled if archive_library is+ * empty or if archive_library is set to "shell".+ */+#define ShellArchivingEnabled() \+ (XLogArchiveLibrary[0] == '\0' || strcmp(XLogArchiveLibrary, "shell") == 0)
#endif /* _PGARCH_H */
--
2.16.6
[application/octet-stream] v12-0003-Add-test-archive-module.patch (9.2K, ../3842CF04-528B-42A5-A12D-BA3970EF1E53@amazon.com/4-v12-0003-Add-test-archive-module.patch)
download | inline diff:
From 7936c8ff9faff34ca5e7de85f4092480c8cb11c9 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Fri, 19 Nov 2021 01:05:43 +0000
Subject: [PATCH v12 3/4] Add test archive module.
---
src/test/modules/Makefile | 1 +
src/test/modules/basic_archive/.gitignore | 4 +
src/test/modules/basic_archive/Makefile | 20 +++
src/test/modules/basic_archive/basic_archive.c | 185 +++++++++++++++++++++
src/test/modules/basic_archive/basic_archive.conf | 3 +
.../basic_archive/expected/basic_archive.out | 29 ++++
.../modules/basic_archive/sql/basic_archive.sql | 22 +++
7 files changed, 264 insertions(+)
create mode 100644 src/test/modules/basic_archive/.gitignore
create mode 100644 src/test/modules/basic_archive/Makefile
create mode 100644 src/test/modules/basic_archive/basic_archive.c
create mode 100644 src/test/modules/basic_archive/basic_archive.conf
create mode 100644 src/test/modules/basic_archive/expected/basic_archive.out
create mode 100644 src/test/modules/basic_archive/sql/basic_archive.sql
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefileindex dffc79b2d9..b49e508a2c 100644--- a/src/test/modules/Makefile+++ b/src/test/modules/Makefile@@ -5,6 +5,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
SUBDIRS = \
+ basic_archive \
brin \
commit_ts \
delay_execution \
diff --git a/src/test/modules/basic_archive/.gitignore b/src/test/modules/basic_archive/.gitignorenew file mode 100644index 0000000000..5dcb3ff972--- /dev/null+++ b/src/test/modules/basic_archive/.gitignore@@ -0,0 +1,4 @@+# Generated subdirectories+/log/+/results/+/tmp_check/diff --git a/src/test/modules/basic_archive/Makefile b/src/test/modules/basic_archive/Makefilenew file mode 100644index 0000000000..ffbf846b68--- /dev/null+++ b/src/test/modules/basic_archive/Makefile@@ -0,0 +1,20 @@+# src/test/modules/basic_archive/Makefile++MODULES = basic_archive+PGFILEDESC = "basic_archive - basic archive module"++REGRESS = basic_archive+REGRESS_OPTS = --temp-config $(top_srcdir)/src/test/modules/basic_archive/basic_archive.conf++NO_INSTALLCHECK = 1++ifdef USE_PGXS+PG_CONFIG = pg_config+PGXS := $(shell $(PG_CONFIG) --pgxs)+include $(PGXS)+else+subdir = src/test/modules/basic_archive+top_builddir = ../../../..+include $(top_builddir)/src/Makefile.global+include $(top_srcdir)/contrib/contrib-global.mk+endifdiff --git a/src/test/modules/basic_archive/basic_archive.c b/src/test/modules/basic_archive/basic_archive.cnew file mode 100644index 0000000000..66cbbaa7b5--- /dev/null+++ b/src/test/modules/basic_archive/basic_archive.c@@ -0,0 +1,185 @@+/*-------------------------------------------------------------------------+ *+ * basic_archive.c+ *+ * This file demonstrates a basic archive library implementation that is+ * roughly equivalent to the following shell command:+ *+ * test ! -f /path/to/dest && cp /path/to/src /path/to/dest+ *+ * One notable difference between this module and the shell command above+ * is that this module first copies the file to a temporary destination,+ * syncs it to disk, and then durably moves it to the final destination.+ *+ * Copyright (c) 2022, PostgreSQL Global Development Group+ *+ * IDENTIFICATION+ * src/test/modules/basic_archive/basic_archive.c+ *+ *-------------------------------------------------------------------------+ */+#include "postgres.h"++#include <sys/stat.h>+#include <unistd.h>++#include "miscadmin.h"+#include "postmaster/pgarch.h"+#include "storage/copydir.h"+#include "storage/fd.h"+#include "utils/guc.h"++PG_MODULE_MAGIC;++void _PG_init(void);+void _PG_archive_module_init(ArchiveModuleCallbacks *cb);++static char *archive_directory = NULL;++static bool basic_archive_configured(void);+static bool basic_archive_file(const char *file, const char *path);+static bool check_archive_directory(char **newval, void **extra, GucSource source);++/*+ * _PG_init+ *+ * Defines the module's GUC.+ */+void+_PG_init(void)+{+ DefineCustomStringVariable("basic_archive.archive_directory",+ gettext_noop("Archive file destination directory."),+ NULL,+ &archive_directory,+ "",+ PGC_SIGHUP,+ 0,+ check_archive_directory, NULL, NULL);++ EmitWarningsOnPlaceholders("basic_archive");+}++/*+ * _PG_archive_module_init+ *+ * Returns the module's archiving callbacks.+ */+void+_PG_archive_module_init(ArchiveModuleCallbacks *cb)+{+ AssertVariableIsOfType(&_PG_archive_module_init, ArchiveModuleInit);++ cb->check_configured_cb = basic_archive_configured;+ cb->archive_file_cb = basic_archive_file;+}++/*+ * check_archive_directory+ *+ * Checks that the provided archive directory exists.+ */+static bool+check_archive_directory(char **newval, void **extra, GucSource source)+{+ struct stat st;++ /*+ * The default value is an empty string, so we have to accept that value.+ * Our check_configured callback also checks for this and prevents archiving+ * from proceeding if it is still empty.+ */+ if (*newval == NULL || *newval[0] == '\0')+ return true;++ /*+ * Make sure the file paths won't be too long. The docs indicate that the+ * file names to be archived can be up to 64 characters long.+ */+ if (strlen(*newval) + 64 + 2 >= MAXPGPATH)+ {+ GUC_check_errdetail("archive directory too long");+ return false;+ }++ /*+ * Do a basic sanity check that the specified archive directory exists. It+ * could be removed at some point in the future, so we still need to be+ * prepared for it not to exist in the actual archiving logic.+ */+ if (stat(*newval, &st) != 0 || !S_ISDIR(st.st_mode))+ {+ GUC_check_errdetail("specified archive directory does not exist");+ return false;+ }++ return true;+}++/*+ * basic_archive_configured+ *+ * Checks that archive_directory is not blank.+ */+static bool+basic_archive_configured(void)+{+ return archive_directory != NULL && archive_directory[0] != '\0';+}++/*+ * basic_archive_file+ *+ * Archives one file.+ */+static bool+basic_archive_file(const char *file, const char *path)+{+ char destination[MAXPGPATH];+ char temp[MAXPGPATH];+ struct stat st;++ ereport(DEBUG3,+ (errmsg("archiving \"%s\" via basic_archive", file)));++ snprintf(destination, MAXPGPATH, "%s/%s", archive_directory, file);+ snprintf(temp, MAXPGPATH, "%s/%s", archive_directory, "archtemp");++ /*+ * First, check if the file has already been archived. If the archive file+ * already exists, something might be wrong, so we just fail.+ */+ if (stat(destination, &st) == 0)+ {+ ereport(WARNING,+ (errmsg("archive file \"%s\" already exists", destination)));+ return false;+ }+ else if (errno != ENOENT)+ ereport(ERROR,+ (errcode_for_file_access(),+ errmsg("could not stat file \"%s\": %m", destination)));++ /*+ * Remove pre-existing temporary file, if one exists.+ */+ if (unlink(temp) != 0 && errno != ENOENT)+ ereport(ERROR,+ (errcode_for_file_access(),+ errmsg("could not unlink file \"%s\": %m", temp)));++ /*+ * Copy the file to its temporary destination.+ */+ copy_file(unconstify(char *, path), temp);++ /*+ * Sync the temporary file to disk and move it to its final destination.+ */+ (void) durable_rename_excl(temp, destination, ERROR);++ ereport(DEBUG1,+ (errmsg("archived \"%s\" via basic_archive", file)));++ return true;+}diff --git a/src/test/modules/basic_archive/basic_archive.conf b/src/test/modules/basic_archive/basic_archive.confnew file mode 100644index 0000000000..b26b2d4144--- /dev/null+++ b/src/test/modules/basic_archive/basic_archive.conf@@ -0,0 +1,3 @@+archive_mode = 'on'+archive_library = 'basic_archive'+basic_archive.archive_directory = '.'diff --git a/src/test/modules/basic_archive/expected/basic_archive.out b/src/test/modules/basic_archive/expected/basic_archive.outnew file mode 100644index 0000000000..0015053e0f--- /dev/null+++ b/src/test/modules/basic_archive/expected/basic_archive.out@@ -0,0 +1,29 @@+CREATE TABLE test (a INT);+SELECT 1 FROM pg_switch_wal();+ ?column? +----------+ 1+(1 row)++DO $$+DECLARE+ archived bool;+ loops int := 0;+BEGIN+ LOOP+ archived := count(*) > 0 FROM pg_ls_dir('.', false, false) a+ WHERE a ~ '^[0-9A-F]{24}$';+ IF archived OR loops > 120 * 10 THEN EXIT; END IF;+ PERFORM pg_sleep(0.1);+ loops := loops + 1;+ END LOOP;+END+$$;+SELECT count(*) > 0 FROM pg_ls_dir('.', false, false) a+ WHERE a ~ '^[0-9A-F]{24}$';+ ?column? +----------+ t+(1 row)++DROP TABLE test;diff --git a/src/test/modules/basic_archive/sql/basic_archive.sql b/src/test/modules/basic_archive/sql/basic_archive.sqlnew file mode 100644index 0000000000..14e236d57a--- /dev/null+++ b/src/test/modules/basic_archive/sql/basic_archive.sql@@ -0,0 +1,22 @@+CREATE TABLE test (a INT);+SELECT 1 FROM pg_switch_wal();++DO $$+DECLARE+ archived bool;+ loops int := 0;+BEGIN+ LOOP+ archived := count(*) > 0 FROM pg_ls_dir('.', false, false) a+ WHERE a ~ '^[0-9A-F]{24}$';+ IF archived OR loops > 120 * 10 THEN EXIT; END IF;+ PERFORM pg_sleep(0.1);+ loops := loops + 1;+ END LOOP;+END+$$;++SELECT count(*) > 0 FROM pg_ls_dir('.', false, false) a+ WHERE a ~ '^[0-9A-F]{24}$';++DROP TABLE test;--
2.16.6
[application/octet-stream] v12-0004-Add-documentation-for-archive-modules.patch (26.4K, ../3842CF04-528B-42A5-A12D-BA3970EF1E53@amazon.com/5-v12-0004-Add-documentation-for-archive-modules.patch)
download | inline diff:
From 23ab54853e1ed3b5533f1017b159469268e771ea Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Fri, 19 Nov 2021 01:06:01 +0000
Subject: [PATCH v12 4/4] Add documentation for archive modules.
---
doc/src/sgml/archive-modules.sgml | 123 ++++++++++++++++++++++++++++++++++++
doc/src/sgml/backup.sgml | 83 +++++++++++++++---------
doc/src/sgml/config.sgml | 37 ++++++++---
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/high-availability.sgml | 6 +-
doc/src/sgml/postgres.sgml | 1 +
doc/src/sgml/ref/pg_basebackup.sgml | 4 +-
doc/src/sgml/ref/pg_receivewal.sgml | 6 +-
doc/src/sgml/wal.sgml | 2 +-
9 files changed, 215 insertions(+), 48 deletions(-)
create mode 100644 doc/src/sgml/archive-modules.sgml
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgmlnew file mode 100644index 0000000000..d52aaaf1f5--- /dev/null+++ b/doc/src/sgml/archive-modules.sgml@@ -0,0 +1,123 @@+<!-- doc/src/sgml/archive-modules.sgml -->++<chapter id="archive-modules">+ <title>Archive Modules</title>+ <indexterm zone="archive-modules">+ <primary>Archive Modules</primary>+ </indexterm>++ <para>+ PostgreSQL provides infrastructure to create custom modules for continuous+ archiving (see <xref linkend="continuous-archiving"/>). While archiving via+ a shell command (i.e., <xref linkend="guc-archive-command"/>) is much+ simpler, a custom archive module will often be considerably more robust and+ performant.+ </para>++ <para>+ When a custom <xref linkend="guc-archive-library"/> is configured, PostgreSQL+ will submit completed WAL files to the module, and the server will avoid+ recyling or removing these WAL files until the module indicates that the files+ were successfully archived. It is ultimately up to the module to decide what+ to do with each WAL file, but many recommendations are listed at+ <xref linkend="backup-archiving-wal"/>.+ </para>++ <para>+ Archiving modules must at least consist of an initialization function (see+ <xref linkend="archive-module-init"/>) and the required callbacks (see+ <xref linkend="archive-module-callbacks"/>). However, archive modules are+ also permitted to do much more (e.g., declare GUCs and register background+ workers).+ </para>++ <para>+ The <filename>src/test/modules/basic_archive</filename> module contains a+ working example, which demonstrates some useful techniques.+ </para>++ <warning>+ <para>+ There are considerable robustness and security risks in using archive modules+ because, being written in the <literal>C</literal> language, they have access+ to many server resources. Administrators wishing to enable archive modules+ should exercise extreme caution. Only carefully audited modules should be+ loaded.+ </para>+ </warning>++ <sect1 id="archive-module-init">+ <title>Initialization Functions</title>+ <indexterm zone="archive-module-init">+ <primary>_PG_archive_module_init</primary>+ </indexterm>+ <para>+ An archive library is loaded by dynamically loading a shared library with the+ <xref linkend="guc-archive-library"/>'s name as the library base name. The+ normal library search path is used to locate the library. To provide the+ required archive module callbacks and to indicate that the library is+ actually an archive module, it needs to provide a function named+ <function>_PG_archive_module_init</function>. This function is passed a+ struct that needs to be filled with the callback function pointers for+ individual actions.++<programlisting>+typedef struct ArchiveModuleCallbacks+{+ ArchiveCheckConfiguredCB check_configured_cb;+ ArchiveFileCB archive_file_cb;+} ArchiveModuleCallbacks;+typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);+</programlisting>++ Both callbacks are required.+ </para>+ </sect1>++ <sect1 id="archive-module-callbacks">+ <title>Archive Module Callbacks</title>+ <para>+ The archive callbacks define the actual archiving behavior of the module.+ The server will call them as required to process each individual WAL file.+ </para>++ <sect2 id="archive-module-check">+ <title>Check Callback</title>+ <para>+ The <function>check_configured_cb</function> callback is called to determine+ whether the module is fully configured and ready to accept WAL files.++<programlisting>+typedef bool (*ArchiveCheckConfiguredCB) (void);+</programlisting>++ If <literal>true</literal> is returned, the server will proceed with+ archiving the file by calling the <function>archive_file_cb</function>+ callback. If <literal>false</literal> is returned, archiving will not+ proceed. In the latter case, the server will periodically call this+ function, and archiving will proceed if it eventually returns+ <literal>true</literal>.+ </para>+ </sect2>++ <sect2 id="archive-module-archive">+ <title>Archive Callback</title>+ <para>+ The <function>archive_file_cb</function> callback is called to archive a+ single WAL file.++<programlisting>+typedef bool (*ArchiveFileCB) (const char *file, const char *path);+</programlisting>++ If <literal>true</literal> is returned, the server proceeds as if the file+ was successfully archived, which may include recycling or removing the+ original WAL file. If <literal>false</literal> is returned, the server will+ keep the original WAL file and retry archiving later.+ <literal>file</literal> will contain just the file name of the WAL file to+ archive, while <literal>path</literal> contains the full path of the WAL+ file (including the file name).+ </para>+ </sect2>+ </sect1>+</chapter>diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgmlindex cba32b6eb3..b42f1b3ca7 100644--- a/doc/src/sgml/backup.sgml+++ b/doc/src/sgml/backup.sgml@@ -593,20 +593,23 @@ tar -cf backup.tar /usr/local/pgsql/data
provide the database administrator with flexibility,
<productname>PostgreSQL</productname> tries not to make any assumptions about how
the archiving will be done. Instead, <productname>PostgreSQL</productname> lets
- the administrator specify a shell command to be executed to copy a- completed segment file to wherever it needs to go. The command could be- as simple as a <literal>cp</literal>, or it could invoke a complex shell- script — it's all up to you.+ the administrator specify an archive library to be executed to copy a+ completed segment file to wherever it needs to go. This could be as simple+ as a shell command that uses <literal>cp</literal>, or it could invoke a+ complex C function — it's all up to you.
</para>
<para>
To enable WAL archiving, set the <xref linkend="guc-wal-level"/>
configuration parameter to <literal>replica</literal> or higher,
<xref linkend="guc-archive-mode"/> to <literal>on</literal>,
- and specify the shell command to use in the <xref- linkend="guc-archive-command"/> configuration parameter. In practice+ and specify the library to use in the <xref+ linkend="guc-archive-library"/> configuration parameter. In practice
these settings will always be placed in the
<filename>postgresql.conf</filename> file.
+ One simple way to archive is to set <varname>archive_library</varname> to+ <literal>shell</literal> and to specify a shell command in+ <xref linkend="guc-archive-command"/>.
In <varname>archive_command</varname>,
<literal>%p</literal> is replaced by the path name of the file to
archive, while <literal>%f</literal> is replaced by only the file name.
@@ -631,7 +634,17 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0
</para>
<para>
- The archive command will be executed under the ownership of the same+ Another way to archive is to use a custom archive module as the+ <varname>archive_library</varname>. Since such modules are written in+ <literal>C</literal>, creating your own may require considerably more effort+ than writing a shell command. However, archive modules can be more+ performant than archiving via shell, and they will have access to many+ useful server resources. For more information about archive modules, see+ <xref linkend="archive-modules"/>.+ </para>++ <para>+ The archive library will be executed under the ownership of the same
user that the <productname>PostgreSQL</productname> server is running as. Since
the series of WAL files being archived contains effectively everything
in your database, you will want to be sure that the archived data is
@@ -640,25 +653,31 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0
</para>
<para>
- It is important that the archive command return zero exit status if and- only if it succeeds. Upon getting a zero result,+ It is important that the archive function return <literal>true</literal> if+ and only if it succeeds. If <literal>true</literal> is returned,
<productname>PostgreSQL</productname> will assume that the file has been
- successfully archived, and will remove or recycle it. However, a nonzero- status tells <productname>PostgreSQL</productname> that the file was not archived;- it will try again periodically until it succeeds.+ successfully archived, and will remove or recycle it. However, a return+ value of <literal>false</literal> tells+ <productname>PostgreSQL</productname> that the file was not archived; it+ will try again periodically until it succeeds. If you are archiving via a+ shell command, the appropriate return values can be achieved by returning+ <literal>0</literal> if the command succeeds and a nonzero value if it+ fails.
</para>
<para>
- When the archive command is terminated by a signal (other than- <systemitem>SIGTERM</systemitem> that is used as part of a server- shutdown) or an error by the shell with an exit status greater than- 125 (such as command not found), the archiver process aborts and gets- restarted by the postmaster. In such cases, the failure is- not reported in <xref linkend="pg-stat-archiver-view"/>.+ If the archive function emits an <literal>ERROR</literal> or+ <literal>FATAL</literal>, the archiver process aborts and gets restarted by+ the postmaster. If you are archiving via shell command, FATAL is emitted if+ the command is terminated by a signal (other than+ <systemitem>SIGTERM</systemitem> that is used as part of a server shutdown)+ or an error by the shell with an exit status greater than 125 (such as+ command not found). In such cases, the failure is not reported in+ <xref linkend="pg-stat-archiver-view"/>.
</para>
<para>
- The archive command should generally be designed to refuse to overwrite+ The archive library should generally be designed to refuse to overwrite
any pre-existing archive file. This is an important safety feature to
preserve the integrity of your archive in case of administrator error
(such as sending the output of two different servers to the same archive
@@ -666,9 +685,9 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0
</para>
<para>
- It is advisable to test your proposed archive command to ensure that it+ It is advisable to test your proposed archive library to ensure that it
indeed does not overwrite an existing file, <emphasis>and that it returns
- nonzero status in this case</emphasis>.+ <literal>false</literal> in this case</emphasis>.
The example command above for Unix ensures this by including a separate
<command>test</command> step. On some Unix platforms, <command>cp</command> has
switches such as <option>-i</option> that can be used to do the same thing
@@ -680,7 +699,7 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0
<para>
While designing your archiving setup, consider what will happen if
- the archive command fails repeatedly because some aspect requires+ the archive library fails repeatedly because some aspect requires
operator intervention or the archive runs out of space. For example, this
could occur if you write to tape without an autochanger; when the tape
fills, nothing further can be archived until the tape is swapped.
@@ -695,7 +714,7 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0
</para>
<para>
- The speed of the archiving command is unimportant as long as it can keep up+ The speed of the archive library is unimportant as long as it can keep up
with the average rate at which your server generates WAL data. Normal
operation continues even if the archiving process falls a little behind.
If archiving falls significantly behind, this will increase the amount of
@@ -707,11 +726,11 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0
</para>
<para>
- In writing your archive command, you should assume that the file names to+ In writing your archive library, you should assume that the file names to
be archived can be up to 64 characters long and can contain any
combination of ASCII letters, digits, and dots. It is not necessary to
- preserve the original relative path (<literal>%p</literal>) but it is necessary to- preserve the file name (<literal>%f</literal>).+ preserve the original relative path but it is necessary to preserve the file+ name.
</para>
<para>
@@ -728,7 +747,7 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0
</para>
<para>
- The archive command is only invoked on completed WAL segments. Hence,+ The archive function is only invoked on completed WAL segments. Hence,
if your server generates only little WAL traffic (or has slack periods
where it does so), there could be a long delay between the completion
of a transaction and its safe recording in archive storage. To put
@@ -758,7 +777,8 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0
contain enough information for archive recovery. (Crash recovery is
unaffected.) For this reason, <varname>wal_level</varname> can only be changed at
server start. However, <varname>archive_command</varname> can be changed with a
- configuration file reload. If you wish to temporarily stop archiving,+ configuration file reload. If you are archiving via shell and wish to+ temporarily stop archiving,
one way to do it is to set <varname>archive_command</varname> to the empty
string (<literal>''</literal>).
This will cause WAL files to accumulate in <filename>pg_wal/</filename> until a
@@ -938,11 +958,11 @@ SELECT * FROM pg_stop_backup(false, true);
On a standby, <varname>archive_mode</varname> must be <literal>always</literal> in order
for <function>pg_stop_backup</function> to wait.
Archiving of these files happens automatically since you have
- already configured <varname>archive_command</varname>. In most cases this+ already configured <varname>archive_library</varname>. In most cases this
happens quickly, but you are advised to monitor your archive
system to ensure there are no delays.
If the archive process has fallen behind
- because of failures of the archive command, it will keep retrying+ because of failures of the archive library, it will keep retrying
until the archive succeeds and the backup is complete.
If you wish to place a time limit on the execution of
<function>pg_stop_backup</function>, set an appropriate
@@ -1500,9 +1520,10 @@ restore_command = 'cp /mnt/server/archivedir/%f %p'
To prepare for low level standalone hot backups, make sure
<varname>wal_level</varname> is set to
<literal>replica</literal> or higher, <varname>archive_mode</varname> to
- <literal>on</literal>, and set up an <varname>archive_command</varname> that performs+ <literal>on</literal>, and set up an <varname>archive_library</varname> that performs
archiving only when a <emphasis>switch file</emphasis> exists. For example:
<programlisting>
+archive_library = 'shell'
archive_command = 'test ! -f /var/lib/pgsql/backup_in_progress || (test ! -f /var/lib/pgsql/archive/%f && cp %p /var/lib/pgsql/archive/%f)'
</programlisting>
This command will perform archiving when
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgmlindex afbb6c35e3..d8b5152930 100644--- a/doc/src/sgml/config.sgml+++ b/doc/src/sgml/config.sgml@@ -3479,7 +3479,7 @@ include_dir 'conf.d'
Maximum size to let the WAL grow during automatic
checkpoints. This is a soft limit; WAL size can exceed
<varname>max_wal_size</varname> under special circumstances, such as
- heavy load, a failing <varname>archive_command</varname>, or a high+ heavy load, a failing <varname>archive_library</varname>, or a high
<varname>wal_keep_size</varname> setting.
If this value is specified without units, it is taken as megabytes.
The default is 1 GB.
@@ -3528,7 +3528,7 @@ include_dir 'conf.d'
<para>
When <varname>archive_mode</varname> is enabled, completed WAL segments
are sent to archive storage by setting
- <xref linkend="guc-archive-command"/>. In addition to <literal>off</literal>,+ <xref linkend="guc-archive-library"/>. In addition to <literal>off</literal>,
to disable, there are two modes: <literal>on</literal>, and
<literal>always</literal>. During normal operation, there is no
difference between the two modes, but when set to <literal>always</literal>
@@ -3538,9 +3538,6 @@ include_dir 'conf.d'
<xref linkend="continuous-archiving-in-standby"/> for details.
</para>
<para>
- <varname>archive_mode</varname> and <varname>archive_command</varname> are- separate variables so that <varname>archive_command</varname> can be- changed without leaving archiving mode.
This parameter can only be set at server start.
<varname>archive_mode</varname> cannot be enabled when
<varname>wal_level</varname> is set to <literal>minimal</literal>.
@@ -3548,6 +3545,28 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-archive-library" xreflabel="archive_library">+ <term><varname>archive_library</varname> (<type>string</type>)+ <indexterm>+ <primary><varname>archive_library</varname> configuration parameter</primary>+ </indexterm>+ </term>+ <listitem>+ <para>+ The library to use for archiving completed WAL file segments. If set to+ <literal>shell</literal> (the default) or an empty string, archiving via+ shell is enabled, and <xref linkend="guc-archive-command"/> is used.+ Otherwise, the specified shared library is used for archiving. For more+ information, see <xref linkend="backup-archiving-wal"/> and+ <xref linkend="archive-modules"/>.+ </para>+ <para>+ This parameter can only be set in the+ <filename>postgresql.conf</filename> file or on the server command line.+ </para>+ </listitem>+ </varlistentry>+
<varlistentry id="guc-archive-command" xreflabel="archive_command">
<term><varname>archive_command</varname> (<type>string</type>)
<indexterm>
@@ -3570,9 +3589,11 @@ include_dir 'conf.d'
<para>
This parameter can only be set in the <filename>postgresql.conf</filename>
file or on the server command line. It is ignored unless
- <varname>archive_mode</varname> was enabled at server start.+ <varname>archive_mode</varname> was enabled at server start and+ <varname>archive_library</varname> specifies to archive via shell command.
If <varname>archive_command</varname> is an empty string (the default) while
- <varname>archive_mode</varname> is enabled, WAL archiving is temporarily+ <varname>archive_mode</varname> is enabled and <varname>archive_library</varname>+ specifies archiving via shell, WAL archiving is temporarily
disabled, but the server continues to accumulate WAL segment files in
the expectation that a command will soon be provided. Setting
<varname>archive_command</varname> to a command that does nothing but
@@ -3592,7 +3613,7 @@ include_dir 'conf.d'
</term>
<listitem>
<para>
- The <xref linkend="guc-archive-command"/> is only invoked for+ The <xref linkend="guc-archive-library"/> is only invoked for
completed WAL segments. Hence, if your server generates little WAL
traffic (or has slack periods where it does so), there could be a
long delay between the completion of a transaction and its safe
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgmlindex 89454e99b9..e6b472ec32 100644--- a/doc/src/sgml/filelist.sgml+++ b/doc/src/sgml/filelist.sgml@@ -99,6 +99,7 @@
<!ENTITY custom-scan SYSTEM "custom-scan.sgml">
<!ENTITY logicaldecoding SYSTEM "logicaldecoding.sgml">
<!ENTITY replication-origins SYSTEM "replication-origins.sgml">
+<!ENTITY archive-modules SYSTEM "archive-modules.sgml">
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgmlindex a265409f02..437712762a 100644--- a/doc/src/sgml/high-availability.sgml+++ b/doc/src/sgml/high-availability.sgml@@ -935,7 +935,7 @@ primary_conninfo = 'host=192.168.1.50 port=5432 user=foo password=foopass'
In lieu of using replication slots, it is possible to prevent the removal
of old WAL segments using <xref linkend="guc-wal-keep-size"/>, or by
storing the segments in an archive using
- <xref linkend="guc-archive-command"/>.+ <xref linkend="guc-archive-library"/>.
However, these methods often result in retaining more WAL segments than
required, whereas replication slots retain only the number of segments
known to be needed. On the other hand, replication slots can retain so
@@ -1386,10 +1386,10 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)'
to <literal>always</literal>, and the standby will call the archive
command for every WAL segment it receives, whether it's by restoring
from the archive or by streaming replication. The shared archive can
- be handled similarly, but the <varname>archive_command</varname> must+ be handled similarly, but the <varname>archive_library</varname> must
test if the file being archived exists already, and if the existing file
has identical contents. This requires more care in the
- <varname>archive_command</varname>, as it must+ <varname>archive_library</varname>, as it must
be careful to not overwrite an existing file with different contents,
but return success if the exactly same file is archived twice. And
all that must be done free of race conditions, if two servers attempt
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgmlindex dba9cf413f..3db6d2160b 100644--- a/doc/src/sgml/postgres.sgml+++ b/doc/src/sgml/postgres.sgml@@ -233,6 +233,7 @@ break is not needed in a wider output rendering.
&bgworker;
&logicaldecoding;
&replication-origins;
+ &archive-modules;
</part>
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgmlindex 9e6807b457..2aaeaca766 100644--- a/doc/src/sgml/ref/pg_basebackup.sgml+++ b/doc/src/sgml/ref/pg_basebackup.sgml@@ -102,8 +102,8 @@ PostgreSQL documentation
<para>
All WAL records required for the backup must contain sufficient full-page writes,
which requires you to enable <varname>full_page_writes</varname> on the primary and
- not to use a tool like <application>pg_compresslog</application> as- <varname>archive_command</varname> to remove full-page writes from WAL files.+ not to use a tool in your <varname>archive_library</varname> to remove+ full-page writes from WAL files.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgmlindex 5de80f8c64..a6b6ba91fb 100644--- a/doc/src/sgml/ref/pg_receivewal.sgml+++ b/doc/src/sgml/ref/pg_receivewal.sgml@@ -40,7 +40,7 @@ PostgreSQL documentation
<para>
<application>pg_receivewal</application> streams the write-ahead
log in real time as it's being generated on the server, and does not wait
- for segments to complete like <xref linkend="guc-archive-command"/> does.+ for segments to complete like <xref linkend="guc-archive-library"/> does.
For this reason, it is not necessary to set
<xref linkend="guc-archive-timeout"/> when using
<application>pg_receivewal</application>.
@@ -488,11 +488,11 @@ PostgreSQL documentation
<para>
When using <application>pg_receivewal</application> instead of
- <xref linkend="guc-archive-command"/> as the main WAL backup method, it is+ <xref linkend="guc-archive-library"/> as the main WAL backup method, it is
strongly recommended to use replication slots. Otherwise, the server is
free to recycle or remove write-ahead log files before they are backed up,
because it does not have any information, either
- from <xref linkend="guc-archive-command"/> or the replication slots, about+ from <xref linkend="guc-archive-library"/> or the replication slots, about
how far the WAL stream has been archived. Note, however, that a
replication slot will fill up the server's disk space if the receiver does
not keep up with fetching the WAL data.
diff --git a/doc/src/sgml/wal.sgml b/doc/src/sgml/wal.sgmlindex 24e1c89503..2bb27a8468 100644--- a/doc/src/sgml/wal.sgml+++ b/doc/src/sgml/wal.sgml@@ -636,7 +636,7 @@
WAL files plus one additional WAL file are
kept at all times. Also, if WAL archiving is used, old segments cannot be
removed or recycled until they are archived. If WAL archiving cannot keep up
- with the pace that WAL is generated, or if <varname>archive_command</varname>+ with the pace that WAL is generated, or if <varname>archive_library</varname>
fails repeatedly, old WAL files will accumulate in <filename>pg_wal</filename>
until the situation is resolved. A slow or failed standby server that
uses a replication slot will have the same effect (see
--
2.16.6
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: bossartn@amazon.com, david@pgmasters.net, masao.fujii@oss.nttdata.com, robertmhaas@gmail.com, michael@paquier.xyz, magnus@hagander.net, sfrost@snowman.net
Subject: Re: archive modules
In-Reply-To: <3842CF04-528B-42A5-A12D-BA3970EF1E53@amazon.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox