Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nDug1-0000un-DN for pgsql-hackers@arkaria.postgresql.org; Sat, 29 Jan 2022 20:50:29 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1nDug0-0007b4-7r for pgsql-hackers@arkaria.postgresql.org; Sat, 29 Jan 2022 20:50:28 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nDufz-0007av-QF for pgsql-hackers@lists.postgresql.org; Sat, 29 Jan 2022 20:50:27 +0000 Received: from mail-pj1-x102c.google.com ([2607:f8b0:4864:20::102c]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1nDufv-0001KF-IM for pgsql-hackers@postgresql.org; Sat, 29 Jan 2022 20:50:27 +0000 Received: by mail-pj1-x102c.google.com with SMTP id q63so9836536pja.1 for ; Sat, 29 Jan 2022 12:50:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=l+VoUzuUb1eJTI6Yyshjy0irGN9IhsdcOrwCCgqzUnU=; b=hm+hU4Yni2cwSxoAH7yoSTTHyrphhjlDuqn/gP83LliAsG7+GTsZYeVl7wULT82+09 QCNHv04aXbDriDUW2hAyOgkPzwerzX80WQDuGLQFqbu6Y80bPy/eqIISZJYr5+orWkQd 7dzDjlp+uyPhYeF8IXN9UjUW8bRpdeFDymfSI+h6n897hMf1Wctsw2/9yDRNgcno3FbF qM6fwM3pDE+T9jybTU2za3BVMr6jCRNVcg8vHqR5gW28RF2i27nqIlNuLSUNVuwPyao7 LY35CzsqaMtBoB42mt8Gi88c81MRjESl2R/QtQsAnqyXSkOsUGwmCBV1GKv5wek/C6v+ x1KQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=l+VoUzuUb1eJTI6Yyshjy0irGN9IhsdcOrwCCgqzUnU=; b=zq5QkiSOCOQk/Cx/5X7QCE5hijJn20fYqrvVcXnbE425Zuw+rvrdfADaH4k39q/Nwc 0nsUM4xO+Prx3J+GikPBKJ5jEq2Xt+uJzzvUg1A+40Eu9P9PrV1ZF8NHaGxRyQEQ6cDN Dzfp2HVGmv18XJF9Ut+ReqkJDHpmZkHK5ZqcY8U3JnS2jg67jv25HIH2gkoRCQf5Lrhq 8u0AcQh2D+xc+bPzYhu9wPa1+u9NnS6CqHiUSmHXwcCP5/XnmqBfoO2QKjoAGFGxRCLV bAc5ErLvro+tlYcE/8vnKXTQxLqQRdbhTHgiYjzxfROqq2KN7JopIuGCQJLyZCZrMNij rngg== X-Gm-Message-State: AOAM530R/KbF8dCDNZmODFjiopHVuE7Gbr7ZWmcdCW+IEUNyB3XZWYgD 45wUyNm1DMpOz047/vcRRxY= X-Google-Smtp-Source: ABdhPJy2MgnnmZngkUbI39HCmVHl/YNTgLQLWndqFmfzcSczpaknIAlB9YeGhquYcIBo3M0IxxMurw== X-Received: by 2002:a17:903:120c:: with SMTP id l12mr14624729plh.135.1643489421289; Sat, 29 Jan 2022 12:50:21 -0800 (PST) Received: from nathanxps13 ([50.54.155.70]) by smtp.gmail.com with ESMTPSA id k21sm13602712pff.33.2022.01.29.12.50.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jan 2022 12:50:20 -0800 (PST) Date: Sat, 29 Jan 2022 12:50:18 -0800 From: Nathan Bossart To: Robert Haas Cc: "Bossart, Nathan" , David Steele , Fujii Masao , "pgsql-hackers@postgresql.org" , Michael Paquier , Magnus Hagander , Stephen Frost Subject: Re: archive modules Message-ID: <20220129205018.GA378582@nathanxps13> References: <316CAF77-A0AC-49AE-A175-1C1ECF6C1981@amazon.com> <71E7A5D1-E418-4516-A21D-BD838BA759CE@amazon.com> <0CF39706-6952-49B1-987D-F012CDF5EEC4@amazon.com> <90D3CEE4-00C1-450C-80B1-88B69D9318DE@amazon.com> <3842CF04-528B-42A5-A12D-BA3970EF1E53@amazon.com> <5D92CC59-47C8-4463-9150-44033DDDD0D8@amazon.com> <20220128200138.GA3398@nathanxps13> <20220128233935.GA6472@nathanxps13> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="zhXaljGHf11kAtnf" Content-Disposition: inline In-Reply-To: <20220128233935.GA6472@nathanxps13> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --zhXaljGHf11kAtnf Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Here is a new revision. I've moved basic_archive to contrib, hardened it as suggested, and added shutdown support for archive modules. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com --zhXaljGHf11kAtnf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v14-0001-Introduce-archive-modules-infrastructure.patch" From f62fea53b93ba7181dfe084b4100eba59eb82aaa Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Fri, 19 Nov 2021 01:04:41 +0000 Subject: [PATCH v14 1/3] Introduce archive modules infrastructure. --- src/backend/access/transam/xlog.c | 2 +- src/backend/postmaster/pgarch.c | 93 +++++++++++++++++-- src/backend/postmaster/shell_archive.c | 24 ++++- src/backend/utils/init/miscinit.c | 1 + src/backend/utils/misc/guc.c | 12 ++- src/backend/utils/misc/postgresql.conf.sample | 1 + src/include/access/xlog.h | 1 - src/include/postmaster/pgarch.h | 52 ++++++++++- 8 files changed, 172 insertions(+), 14 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index dfe2a0bcce..958220c495 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -8831,7 +8831,7 @@ ShutdownXLOG(int code, Datum arg) * process one more time at the end of shutdown). The checkpoint * record will go to the next XLOG file and won't be archived (yet). */ - if (XLogArchivingActive() && XLogArchiveCommandSet()) + if (XLogArchivingActive()) RequestXLogSwitch(false); CreateCheckPoint(CHECKPOINT_IS_SHUTDOWN | CHECKPOINT_IMMEDIATE); diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c index 6e3fcedc97..d4a7ca97ca 100644 --- a/src/backend/postmaster/pgarch.c +++ b/src/backend/postmaster/pgarch.c @@ -89,6 +89,8 @@ typedef struct PgArchData slock_t arch_lck; } PgArchData; +char *XLogArchiveLibrary = ""; + /* ---------- * Local data @@ -96,6 +98,8 @@ typedef struct PgArchData */ static time_t last_sigterm_time = 0; static PgArchData *PgArch = NULL; +static ArchiveModuleCallbacks ArchiveContext; + /* * Stuff for tracking multiple files to archive from each scan of @@ -140,6 +144,7 @@ static void pgarch_archiveDone(char *xlog); static void pgarch_die(int code, Datum arg); static void HandlePgArchInterrupts(void); static int ready_file_comparator(Datum a, Datum b, void *arg); +static void LoadArchiveLibrary(void); /* Report shared memory space needed by PgArchShmemInit */ Size @@ -236,6 +241,11 @@ PgArchiverMain(void) */ PgArch->pgprocno = MyProc->pgprocno; + /* + * Load the archive_library. + */ + LoadArchiveLibrary(); + /* Create workspace for pgarch_readyXlog() */ arch_files = palloc(sizeof(struct arch_files_state)); arch_files->arch_files_size = 0; @@ -407,11 +417,12 @@ pgarch_ArchiverCopyLoop(void) */ HandlePgArchInterrupts(); - /* can't do anything if no command ... */ - if (!XLogArchiveCommandSet()) + /* can't do anything if not configured ... */ + if (ArchiveContext.check_configured_cb != NULL && + !ArchiveContext.check_configured_cb()) { ereport(WARNING, - (errmsg("archive_mode enabled, yet archive_command is not set"))); + (errmsg("archive_mode enabled, yet archiving is not configured"))); return; } @@ -492,7 +503,7 @@ pgarch_ArchiverCopyLoop(void) /* * pgarch_archiveXlog * - * Invokes system(3) to copy one archive file to wherever it should go + * Invokes archive_file_cb to copy one archive file to wherever it should go * * Returns true if successful */ @@ -509,7 +520,7 @@ pgarch_archiveXlog(char *xlog) snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog); set_ps_display(activitymsg); - ret = shell_archive_file(xlog, pathname); + ret = ArchiveContext.archive_file_cb(xlog, pathname); if (ret) snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog); else @@ -759,13 +770,79 @@ HandlePgArchInterrupts(void) if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); + /* Perform logging of memory contexts of this process */ + if (LogMemoryContextPending) + ProcessLogMemoryContextInterrupt(); + if (ConfigReloadPending) { + char *archiveLib = pstrdup(XLogArchiveLibrary); + bool archiveLibChanged; + ConfigReloadPending = false; ProcessConfigFile(PGC_SIGHUP); + + archiveLibChanged = strcmp(XLogArchiveLibrary, archiveLib) != 0; + pfree(archiveLib); + + if (archiveLibChanged) + { + /* + * Call the currently loaded archive module's shutdown callback, if + * one is defined. + */ + if (ArchiveContext.shutdown_cb != NULL) + ArchiveContext.shutdown_cb(); + + /* + * Ideally, we would simply unload the previous archive module and + * load the new one, but there is presently no mechanism for + * unloading a library (see the comment above + * internal_unload_library()). To deal with this, we simply restart + * the archiver. The new archive module will be loaded when the new + * archiver process starts up. + */ + ereport(LOG, + (errmsg("restarting archiver process because value of " + "\"archive_library\" was changed"))); + + proc_exit(0); + } } +} - /* Perform logging of memory contexts of this process */ - if (LogMemoryContextPending) - ProcessLogMemoryContextInterrupt(); +/* + * LoadArchiveLibrary + * + * Loads the archiving callbacks into our local ArchiveContext. + */ +static void +LoadArchiveLibrary(void) +{ + ArchiveModuleInit archive_init; + + memset(&ArchiveContext, 0, sizeof(ArchiveModuleCallbacks)); + + /* + * If shell archiving is enabled, use our special initialization + * function. Otherwise, load the library and call its + * _PG_archive_module_init(). + */ + if (ShellArchivingEnabled()) + archive_init = shell_archive_init; + else + archive_init = (ArchiveModuleInit) + load_external_function(XLogArchiveLibrary, + "_PG_archive_module_init", false, NULL); + + if (archive_init == NULL) + ereport(ERROR, + (errmsg("archive modules have to declare the " + "_PG_archive_module_init symbol"))); + + (*archive_init) (&ArchiveContext); + + if (ArchiveContext.archive_file_cb == NULL) + ereport(ERROR, + (errmsg("archive modules must register an archive callback"))); } diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/postmaster/shell_archive.c index b54e701da4..19e240c205 100644 --- a/src/backend/postmaster/shell_archive.c +++ b/src/backend/postmaster/shell_archive.c @@ -2,6 +2,10 @@ * * shell_archive.c * + * This archiving function uses a user-specified shell command (the + * archive_command GUC) to copy write-ahead log files. It is used as the + * default, but other modules may define their own custom archiving logic. + * * Copyright (c) 2022, PostgreSQL Global Development Group * * IDENTIFICATION @@ -17,7 +21,25 @@ #include "pgstat.h" #include "postmaster/pgarch.h" -bool +static bool shell_archive_configured(void); +static bool shell_archive_file(const char *file, const char *path); + +void +shell_archive_init(ArchiveModuleCallbacks *cb) +{ + AssertVariableIsOfType(&shell_archive_init, ArchiveModuleInit); + + cb->check_configured_cb = shell_archive_configured; + cb->archive_file_cb = shell_archive_file; +} + +static bool +shell_archive_configured(void) +{ + return XLogArchiveCommand[0] != '\0'; +} + +static bool shell_archive_file(const char *file, const char *path) { char xlogarchcmd[MAXPGPATH]; diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c index 0f2570d626..0868e5a24f 100644 --- a/src/backend/utils/init/miscinit.c +++ b/src/backend/utils/init/miscinit.c @@ -38,6 +38,7 @@ #include "pgstat.h" #include "postmaster/autovacuum.h" #include "postmaster/interrupt.h" +#include "postmaster/pgarch.h" #include "postmaster/postmaster.h" #include "storage/fd.h" #include "storage/ipc.h" diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 4c94f09c64..86b223821f 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -3881,13 +3881,23 @@ static struct config_string ConfigureNamesString[] = { {"archive_command", PGC_SIGHUP, WAL_ARCHIVING, gettext_noop("Sets the shell command that will be called to archive a WAL file."), - NULL + gettext_noop("This is unused if \"archive_library\" does not indicate archiving via shell is enabled.") }, &XLogArchiveCommand, "", NULL, NULL, show_archive_command }, + { + {"archive_library", PGC_SIGHUP, WAL_ARCHIVING, + gettext_noop("Sets the library that will be called to archive a WAL file."), + gettext_noop("A value of \"shell\" or an empty string indicates that \"archive_command\" should be used.") + }, + &XLogArchiveLibrary, + "shell", + NULL, NULL, NULL + }, + { {"restore_command", PGC_SIGHUP, WAL_ARCHIVE_RECOVERY, gettext_noop("Sets the shell command that will be called to retrieve an archived WAL file."), diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 817d5f5324..b4376d76aa 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -245,6 +245,7 @@ #archive_mode = off # enables archiving; off, on, or always # (change requires restart) +#archive_library = 'shell' # library to use to archive a logfile segment #archive_command = '' # command to use to archive a logfile segment # placeholders: %p = path of file to archive # %f = file name only diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index bb0c52686a..85114b2e5f 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -155,7 +155,6 @@ extern PGDLLIMPORT int wal_level; /* Is WAL archiving enabled always (even during recovery)? */ #define XLogArchivingAlways() \ (AssertMacro(XLogArchiveMode == ARCHIVE_MODE_OFF || wal_level >= WAL_LEVEL_REPLICA), XLogArchiveMode == ARCHIVE_MODE_ALWAYS) -#define XLogArchiveCommandSet() (XLogArchiveCommand[0] != '\0') /* * Is WAL-logging necessary for archival or log-shipping, or can we skip diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h index 991a6d0616..732b12c0ba 100644 --- a/src/include/postmaster/pgarch.h +++ b/src/include/postmaster/pgarch.h @@ -33,7 +33,55 @@ extern void PgArchiverMain(void) pg_attribute_noreturn(); extern void PgArchWakeup(void); extern void PgArchForceDirScan(void); -/* in shell_archive.c */ -extern bool shell_archive_file(const char *file, const char *path); +/* + * The value of the archive_library GUC. + */ +extern char *XLogArchiveLibrary; + +/* + * Callback that gets called to determine if the archive module is + * configured. + */ +typedef bool (*ArchiveCheckConfiguredCB) (void); + +/* + * Callback called to archive a single WAL file. + */ +typedef bool (*ArchiveFileCB) (const char *file, const char *path); + +/* + * Called to shutdown an archive module. + */ +typedef void (*ArchiveShutdownCB) (void); + +/* + * Archive module callbacks + */ +typedef struct ArchiveModuleCallbacks +{ + ArchiveCheckConfiguredCB check_configured_cb; + ArchiveFileCB archive_file_cb; + ArchiveShutdownCB shutdown_cb; +} ArchiveModuleCallbacks; + +/* + * Type of the shared library symbol _PG_archive_module_init that is looked + * up when loading an archive library. + */ +typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb); + +/* + * Since the logic for archiving via a shell command is in the core server + * and does not need to be loaded via a shared library, it has a special + * initialization function. + */ +extern void shell_archive_init(ArchiveModuleCallbacks *cb); + +/* + * We consider archiving via shell to be enabled if archive_library is + * empty or if archive_library is set to "shell". + */ +#define ShellArchivingEnabled() \ + (XLogArchiveLibrary[0] == '\0' || strcmp(XLogArchiveLibrary, "shell") == 0) #endif /* _PGARCH_H */ -- 2.25.1 --zhXaljGHf11kAtnf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v14-0002-Add-test-archive-module.patch" From cd4844b313f006111a94186b5039a48f5960dac1 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Fri, 19 Nov 2021 01:05:43 +0000 Subject: [PATCH v14 2/3] Add test archive module. --- contrib/Makefile | 1 + contrib/basic_archive/.gitignore | 4 + contrib/basic_archive/Makefile | 20 ++ contrib/basic_archive/basic_archive.c | 287 ++++++++++++++++++ contrib/basic_archive/basic_archive.conf | 3 + .../basic_archive/expected/basic_archive.out | 29 ++ contrib/basic_archive/sql/basic_archive.sql | 22 ++ 7 files changed, 366 insertions(+) create mode 100644 contrib/basic_archive/.gitignore create mode 100644 contrib/basic_archive/Makefile create mode 100644 contrib/basic_archive/basic_archive.c create mode 100644 contrib/basic_archive/basic_archive.conf create mode 100644 contrib/basic_archive/expected/basic_archive.out create mode 100644 contrib/basic_archive/sql/basic_archive.sql diff --git a/contrib/Makefile b/contrib/Makefile index 87bf87ab90..e3e221308b 100644 --- a/contrib/Makefile +++ b/contrib/Makefile @@ -9,6 +9,7 @@ SUBDIRS = \ amcheck \ auth_delay \ auto_explain \ + basic_archive \ bloom \ btree_gin \ btree_gist \ diff --git a/contrib/basic_archive/.gitignore b/contrib/basic_archive/.gitignore new file mode 100644 index 0000000000..5dcb3ff972 --- /dev/null +++ b/contrib/basic_archive/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/contrib/basic_archive/Makefile b/contrib/basic_archive/Makefile new file mode 100644 index 0000000000..14d036e1c4 --- /dev/null +++ b/contrib/basic_archive/Makefile @@ -0,0 +1,20 @@ +# contrib/basic_archive/Makefile + +MODULES = basic_archive +PGFILEDESC = "basic_archive - basic archive module" + +REGRESS = basic_archive +REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/basic_archive/basic_archive.conf + +NO_INSTALLCHECK = 1 + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = contrib/basic_archive +top_builddir = ../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c new file mode 100644 index 0000000000..ac6d75f51c --- /dev/null +++ b/contrib/basic_archive/basic_archive.c @@ -0,0 +1,287 @@ +/*------------------------------------------------------------------------- + * + * basic_archive.c + * + * This file demonstrates a basic archive library implementation that is + * roughly equivalent to the following shell command: + * + * test ! -f /path/to/dest && cp /path/to/src /path/to/dest + * + * One notable difference between this module and the shell command above + * is that this module first copies the file to a temporary destination, + * syncs it to disk, and then durably moves it to the final destination. + * + * Copyright (c) 2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * contrib/basic_archive/basic_archive.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include +#include + +#include "miscadmin.h" +#include "postmaster/pgarch.h" +#include "storage/copydir.h" +#include "storage/fd.h" +#include "utils/guc.h" + +PG_MODULE_MAGIC; + +void _PG_init(void); +void _PG_archive_module_init(ArchiveModuleCallbacks *cb); + +static char *archive_directory = NULL; + +static bool basic_archive_configured(void); +static bool basic_archive_file(const char *file, const char *path); +static bool check_archive_directory(char **newval, void **extra, GucSource source); +static bool compare_files(const char *file1, const char *file2); + +/* + * _PG_init + * + * Defines the module's GUC. + */ +void +_PG_init(void) +{ + DefineCustomStringVariable("basic_archive.archive_directory", + gettext_noop("Archive file destination directory."), + NULL, + &archive_directory, + "", + PGC_SIGHUP, + 0, + check_archive_directory, NULL, NULL); + + EmitWarningsOnPlaceholders("basic_archive"); +} + +/* + * _PG_archive_module_init + * + * Returns the module's archiving callbacks. + */ +void +_PG_archive_module_init(ArchiveModuleCallbacks *cb) +{ + AssertVariableIsOfType(&_PG_archive_module_init, ArchiveModuleInit); + + cb->check_configured_cb = basic_archive_configured; + cb->archive_file_cb = basic_archive_file; +} + +/* + * check_archive_directory + * + * Checks that the provided archive directory exists. + */ +static bool +check_archive_directory(char **newval, void **extra, GucSource source) +{ + struct stat st; + + /* + * The default value is an empty string, so we have to accept that value. + * Our check_configured callback also checks for this and prevents archiving + * from proceeding if it is still empty. + */ + if (*newval == NULL || *newval[0] == '\0') + return true; + + /* + * Make sure the file paths won't be too long. The docs indicate that the + * file names to be archived can be up to 64 characters long. + */ + if (strlen(*newval) + 64 + 2 >= MAXPGPATH) + { + GUC_check_errdetail("archive directory too long"); + return false; + } + + /* + * Do a basic sanity check that the specified archive directory exists. It + * could be removed at some point in the future, so we still need to be + * prepared for it not to exist in the actual archiving logic. + */ + if (stat(*newval, &st) != 0 || !S_ISDIR(st.st_mode)) + { + GUC_check_errdetail("specified archive directory does not exist"); + return false; + } + + return true; +} + +/* + * basic_archive_configured + * + * Checks that archive_directory is not blank. + */ +static bool +basic_archive_configured(void) +{ + return archive_directory != NULL && archive_directory[0] != '\0'; +} + +/* + * basic_archive_file + * + * Archives one file. + */ +static bool +basic_archive_file(const char *file, const char *path) +{ + char destination[MAXPGPATH]; + char temp[MAXPGPATH + 64]; + struct stat st; + + ereport(DEBUG3, + (errmsg("archiving \"%s\" via basic_archive", file))); + + snprintf(destination, MAXPGPATH, "%s/%s", archive_directory, file); + + /* + * First, check if the file has already been archived. If it already exists + * and has the same contents as the file we're trying to archive, we can + * return success (after ensuring the file is persisted to disk). This + * scenario is possible if the server crashed after archiving the file but + * before renaming its .ready file to .done. + * + * If the archive file already exists but has different contents, something + * might be wrong, so we just fail. + */ + if (stat(destination, &st) == 0) + { + if (compare_files(path, destination)) + { + ereport(DEBUG3, + (errmsg("archive file \"%s\" already exists with identical contents", + destination))); + + fsync_fname(destination, false); + fsync_fname(archive_directory, true); + + return true; + } + + ereport(WARNING, + (errmsg("archive file \"%s\" already exists", destination))); + return false; + } + else if (errno != ENOENT) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not stat file \"%s\": %m", destination))); + + /* + * Pick a sufficiently random name for the temporary file so that a + * collision is unlikely. This helps avoid problems in case a temporary + * file was left around after a crash or another server happens to be + * archiving to the same directory. + */ + snprintf(temp, sizeof(temp), "%s/%s.%s.%d.%d", archive_directory, + "archtemp", file, MyProcPid, (int) (random() & 0x7fff)); + + /* + * Copy the file to its temporary destination. Note that this will fail if + * temp already exists. + */ + copy_file(unconstify(char *, path), temp); + + /* + * Sync the temporary file to disk and move it to its final destination. + * This will fail if destination already exists. + */ + (void) durable_rename_excl(temp, destination, ERROR); + + ereport(DEBUG1, + (errmsg("archived \"%s\" via basic_archive", file))); + + return true; +} + +/* + * compare_files + * + * Returns whether the contents of the files are the same. + */ +static bool +compare_files(const char *file1, const char *file2) +{ +#define CMP_BUF_SIZE (4096) + char buf1[CMP_BUF_SIZE]; + char buf2[CMP_BUF_SIZE]; + int fd1; + int fd2; + bool ret = true; + + fd1 = OpenTransientFile(file1, O_RDONLY | PG_BINARY); + if (fd1 < 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not open file \"%s\": %m", file1))); + + fd2 = OpenTransientFile(file2, O_RDONLY | PG_BINARY); + if (fd2 < 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not open file \"%s\": %m", file2))); + + for (;;) + { + int nbytes = 0; + int buf1_len = 0; + int buf2_len = 0; + + while (buf1_len < CMP_BUF_SIZE) + { + nbytes = read(fd1, buf1 + buf1_len, CMP_BUF_SIZE - buf1_len); + if (nbytes < 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not read file \"%s\": %m", file1))); + else if (nbytes == 0) + break; + + buf1_len += nbytes; + } + + while (buf2_len < CMP_BUF_SIZE) + { + nbytes = read(fd2, buf2 + buf2_len, CMP_BUF_SIZE - buf2_len); + if (nbytes < 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not read file \"%s\": %m", file2))); + else if (nbytes == 0) + break; + + buf2_len += nbytes; + } + + if (buf1_len != buf2_len || memcmp(buf1, buf2, buf1_len) != 0) + { + ret = false; + break; + } + else if (buf1_len == 0) + break; + } + + if (CloseTransientFile(fd1) != 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not close file \"%s\": %m", file1))); + + if (CloseTransientFile(fd2) != 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not close file \"%s\": %m", file2))); + + return ret; +} diff --git a/contrib/basic_archive/basic_archive.conf b/contrib/basic_archive/basic_archive.conf new file mode 100644 index 0000000000..b26b2d4144 --- /dev/null +++ b/contrib/basic_archive/basic_archive.conf @@ -0,0 +1,3 @@ +archive_mode = 'on' +archive_library = 'basic_archive' +basic_archive.archive_directory = '.' diff --git a/contrib/basic_archive/expected/basic_archive.out b/contrib/basic_archive/expected/basic_archive.out new file mode 100644 index 0000000000..0015053e0f --- /dev/null +++ b/contrib/basic_archive/expected/basic_archive.out @@ -0,0 +1,29 @@ +CREATE TABLE test (a INT); +SELECT 1 FROM pg_switch_wal(); + ?column? +---------- + 1 +(1 row) + +DO $$ +DECLARE + archived bool; + loops int := 0; +BEGIN + LOOP + archived := count(*) > 0 FROM pg_ls_dir('.', false, false) a + WHERE a ~ '^[0-9A-F]{24}$'; + IF archived OR loops > 120 * 10 THEN EXIT; END IF; + PERFORM pg_sleep(0.1); + loops := loops + 1; + END LOOP; +END +$$; +SELECT count(*) > 0 FROM pg_ls_dir('.', false, false) a + WHERE a ~ '^[0-9A-F]{24}$'; + ?column? +---------- + t +(1 row) + +DROP TABLE test; diff --git a/contrib/basic_archive/sql/basic_archive.sql b/contrib/basic_archive/sql/basic_archive.sql new file mode 100644 index 0000000000..14e236d57a --- /dev/null +++ b/contrib/basic_archive/sql/basic_archive.sql @@ -0,0 +1,22 @@ +CREATE TABLE test (a INT); +SELECT 1 FROM pg_switch_wal(); + +DO $$ +DECLARE + archived bool; + loops int := 0; +BEGIN + LOOP + archived := count(*) > 0 FROM pg_ls_dir('.', false, false) a + WHERE a ~ '^[0-9A-F]{24}$'; + IF archived OR loops > 120 * 10 THEN EXIT; END IF; + PERFORM pg_sleep(0.1); + loops := loops + 1; + END LOOP; +END +$$; + +SELECT count(*) > 0 FROM pg_ls_dir('.', false, false) a + WHERE a ~ '^[0-9A-F]{24}$'; + +DROP TABLE test; -- 2.25.1 --zhXaljGHf11kAtnf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v14-0003-Add-documentation-for-archive-modules.patch" From 4c6f42839ebce326f91d577fb3d3160086ea1e24 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Fri, 19 Nov 2021 01:06:01 +0000 Subject: [PATCH v14 3/3] Add documentation for archive modules. --- doc/src/sgml/archive-modules.sgml | 141 ++++++++++++++++++++++++++++ doc/src/sgml/backup.sgml | 83 ++++++++++------ doc/src/sgml/basic-archive.sgml | 81 ++++++++++++++++ doc/src/sgml/config.sgml | 37 ++++++-- doc/src/sgml/contrib.sgml | 1 + doc/src/sgml/filelist.sgml | 2 + doc/src/sgml/high-availability.sgml | 6 +- doc/src/sgml/postgres.sgml | 1 + doc/src/sgml/ref/pg_basebackup.sgml | 4 +- doc/src/sgml/ref/pg_receivewal.sgml | 6 +- doc/src/sgml/wal.sgml | 2 +- 11 files changed, 316 insertions(+), 48 deletions(-) create mode 100644 doc/src/sgml/archive-modules.sgml create mode 100644 doc/src/sgml/basic-archive.sgml diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml new file mode 100644 index 0000000000..722dde0d42 --- /dev/null +++ b/doc/src/sgml/archive-modules.sgml @@ -0,0 +1,141 @@ + + + + Archive Modules + + Archive Modules + + + + PostgreSQL provides infrastructure to create custom modules for continuous + archiving (see ). While archiving via + a shell command (i.e., ) is much + simpler, a custom archive module will often be considerably more robust and + performant. + + + + When a custom is configured, PostgreSQL + will submit completed WAL files to the module, and the server will avoid + recyling or removing these WAL files until the module indicates that the files + were successfully archived. It is ultimately up to the module to decide what + to do with each WAL file, but many recommendations are listed at + . + + + + Archiving modules must at least consist of an initialization function (see + ) and the required callbacks (see + ). However, archive modules are + also permitted to do much more (e.g., declare GUCs and register background + workers). + + + + The contrib/basic_archive module contains a working + example, which demonstrates some useful techniques. + + + + + There are considerable robustness and security risks in using archive modules + because, being written in the C language, they have access + to many server resources. Administrators wishing to enable archive modules + should exercise extreme caution. Only carefully audited modules should be + loaded. + + + + + Initialization Functions + + _PG_archive_module_init + + + An archive library is loaded by dynamically loading a shared library with the + 's name as the library base name. The + normal library search path is used to locate the library. To provide the + required archive module callbacks and to indicate that the library is + actually an archive module, it needs to provide a function named + _PG_archive_module_init. This function is passed a + struct that needs to be filled with the callback function pointers for + individual actions. + + +typedef struct ArchiveModuleCallbacks +{ + ArchiveCheckConfiguredCB check_configured_cb; + ArchiveFileCB archive_file_cb; + ArchiveShutdownCB shutdown_cb; +} ArchiveModuleCallbacks; +typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb); + + + Only the archive_file_cb callback is required. The + others are optional. + + + + + Archive Module Callbacks + + The archive callbacks define the actual archiving behavior of the module. + The server will call them as required to process each individual WAL file. + + + + Check Callback + + The check_configured_cb callback is called to determine + whether the module is fully configured and ready to accept WAL files. If no + check_configured_cb is defined, the server always + assumes the module is configured. + + +typedef bool (*ArchiveCheckConfiguredCB) (void); + + + If true is returned, the server will proceed with + archiving the file by calling the archive_file_cb + callback. If false is returned, archiving will not + proceed. In the latter case, the server will periodically call this + function, and archiving will proceed if it eventually returns + true. + + + + + Archive Callback + + The archive_file_cb callback is called to archive a + single WAL file. + + +typedef bool (*ArchiveFileCB) (const char *file, const char *path); + + + If true is returned, the server proceeds as if the file + was successfully archived, which may include recycling or removing the + original WAL file. If false is returned, the server will + keep the original WAL file and retry archiving later. + file will contain just the file name of the WAL file to + archive, while path contains the full path of the WAL + file (including the file name). + + + + + Shutdown Callback + + The shutdown_cb callback is called when the value of + changes (and before the new archive + library is loaded). If no shutdown_cb is defined, no + special action is taken before loading the new archive library. + + +typedef void (*ArchiveShutdownCB) (void); + + + + + diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index cba32b6eb3..b42f1b3ca7 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -593,20 +593,23 @@ tar -cf backup.tar /usr/local/pgsql/data provide the database administrator with flexibility, PostgreSQL tries not to make any assumptions about how the archiving will be done. Instead, PostgreSQL lets - the administrator specify a shell command to be executed to copy a - completed segment file to wherever it needs to go. The command could be - as simple as a cp, or it could invoke a complex shell - script — it's all up to you. + the administrator specify an archive library to be executed to copy a + completed segment file to wherever it needs to go. This could be as simple + as a shell command that uses cp, or it could invoke a + complex C function — it's all up to you. To enable WAL archiving, set the configuration parameter to replica or higher, to on, - and specify the shell command to use in the configuration parameter. In practice + and specify the library to use in the configuration parameter. In practice these settings will always be placed in the postgresql.conf file. + One simple way to archive is to set archive_library to + shell and to specify a shell command in + . In archive_command, %p is replaced by the path name of the file to archive, while %f is replaced by only the file name. @@ -631,7 +634,17 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0 - The archive command will be executed under the ownership of the same + Another way to archive is to use a custom archive module as the + archive_library. Since such modules are written in + C, creating your own may require considerably more effort + than writing a shell command. However, archive modules can be more + performant than archiving via shell, and they will have access to many + useful server resources. For more information about archive modules, see + . + + + + The archive library will be executed under the ownership of the same user that the PostgreSQL server is running as. Since the series of WAL files being archived contains effectively everything in your database, you will want to be sure that the archived data is @@ -640,25 +653,31 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0 - It is important that the archive command return zero exit status if and - only if it succeeds. Upon getting a zero result, + It is important that the archive function return true if + and only if it succeeds. If true is returned, PostgreSQL will assume that the file has been - successfully archived, and will remove or recycle it. However, a nonzero - status tells PostgreSQL that the file was not archived; - it will try again periodically until it succeeds. + successfully archived, and will remove or recycle it. However, a return + value of false tells + PostgreSQL that the file was not archived; it + will try again periodically until it succeeds. If you are archiving via a + shell command, the appropriate return values can be achieved by returning + 0 if the command succeeds and a nonzero value if it + fails. - When the archive command is terminated by a signal (other than - SIGTERM that is used as part of a server - shutdown) or an error by the shell with an exit status greater than - 125 (such as command not found), the archiver process aborts and gets - restarted by the postmaster. In such cases, the failure is - not reported in . + If the archive function emits an ERROR or + FATAL, the archiver process aborts and gets restarted by + the postmaster. If you are archiving via shell command, FATAL is emitted if + the command is terminated by a signal (other than + SIGTERM that is used as part of a server shutdown) + or an error by the shell with an exit status greater than 125 (such as + command not found). In such cases, the failure is not reported in + . - The archive command should generally be designed to refuse to overwrite + The archive library should generally be designed to refuse to overwrite any pre-existing archive file. This is an important safety feature to preserve the integrity of your archive in case of administrator error (such as sending the output of two different servers to the same archive @@ -666,9 +685,9 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0 - It is advisable to test your proposed archive command to ensure that it + It is advisable to test your proposed archive library to ensure that it indeed does not overwrite an existing file, and that it returns - nonzero status in this case. + false in this case. The example command above for Unix ensures this by including a separate test step. On some Unix platforms, cp has switches such as that can be used to do the same thing @@ -680,7 +699,7 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0 While designing your archiving setup, consider what will happen if - the archive command fails repeatedly because some aspect requires + the archive library fails repeatedly because some aspect requires operator intervention or the archive runs out of space. For example, this could occur if you write to tape without an autochanger; when the tape fills, nothing further can be archived until the tape is swapped. @@ -695,7 +714,7 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0 - The speed of the archiving command is unimportant as long as it can keep up + The speed of the archive library is unimportant as long as it can keep up with the average rate at which your server generates WAL data. Normal operation continues even if the archiving process falls a little behind. If archiving falls significantly behind, this will increase the amount of @@ -707,11 +726,11 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0 - In writing your archive command, you should assume that the file names to + In writing your archive library, you should assume that the file names to be archived can be up to 64 characters long and can contain any combination of ASCII letters, digits, and dots. It is not necessary to - preserve the original relative path (%p) but it is necessary to - preserve the file name (%f). + preserve the original relative path but it is necessary to preserve the file + name. @@ -728,7 +747,7 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0 - The archive command is only invoked on completed WAL segments. Hence, + The archive function is only invoked on completed WAL segments. Hence, if your server generates only little WAL traffic (or has slack periods where it does so), there could be a long delay between the completion of a transaction and its safe recording in archive storage. To put @@ -758,7 +777,8 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0 contain enough information for archive recovery. (Crash recovery is unaffected.) For this reason, wal_level can only be changed at server start. However, archive_command can be changed with a - configuration file reload. If you wish to temporarily stop archiving, + configuration file reload. If you are archiving via shell and wish to + temporarily stop archiving, one way to do it is to set archive_command to the empty string (''). This will cause WAL files to accumulate in pg_wal/ until a @@ -938,11 +958,11 @@ SELECT * FROM pg_stop_backup(false, true); On a standby, archive_mode must be always in order for pg_stop_backup to wait. Archiving of these files happens automatically since you have - already configured archive_command. In most cases this + already configured archive_library. In most cases this happens quickly, but you are advised to monitor your archive system to ensure there are no delays. If the archive process has fallen behind - because of failures of the archive command, it will keep retrying + because of failures of the archive library, it will keep retrying until the archive succeeds and the backup is complete. If you wish to place a time limit on the execution of pg_stop_backup, set an appropriate @@ -1500,9 +1520,10 @@ restore_command = 'cp /mnt/server/archivedir/%f %p' To prepare for low level standalone hot backups, make sure wal_level is set to replica or higher, archive_mode to - on, and set up an archive_command that performs + on, and set up an archive_library that performs archiving only when a switch file exists. For example: +archive_library = 'shell' archive_command = 'test ! -f /var/lib/pgsql/backup_in_progress || (test ! -f /var/lib/pgsql/archive/%f && cp %p /var/lib/pgsql/archive/%f)' This command will perform archiving when diff --git a/doc/src/sgml/basic-archive.sgml b/doc/src/sgml/basic-archive.sgml new file mode 100644 index 0000000000..0b650f17a8 --- /dev/null +++ b/doc/src/sgml/basic-archive.sgml @@ -0,0 +1,81 @@ + + + + basic_archive + + + basic_archive + + + + basic_archive is an example of an archive module. This + module copies completed WAL segment files to the specified directory. This + may not be especially useful, but it can serve as a starting point for + developing your own archive module. For more information about archive + modules, see . + + + + In order to function, this module must be loaded via + , and + must be enabled. + + + + Configuration Parameters + + + + + basic_archive.archive_directory (string) + + basic_archive.archive_directory configuration parameter + + + + + The directory where the server should copy WAL segment files. This + directory must already exist. The default is an empty string, which + effectively halts WAL archiving, but if + is enabled, the server will accumulate WAL segment files in the + expectation that a value will soon be provided. + + + + + + + These parameters must be set in postgresql.conf. + Typical usage might be: + + + +# postgresql.conf +archive_mode = 'on' +archive_library = 'basic_archive' +basic_archive.archive_directory = '/path/to/archive/directory' + + + + + Notes + + + Server crashes may leave temporary files with the prefix + archtemp in the archive directory. It is recommended to + delete such files before restarting the server after a crash. It is safe to + remove such files while the server is running as long as they are unrelated + to any archiving still in progress, but users should use extra caution when + doing so. + + + + + Author + + + Nathan Bossart + + + + diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 692d8a2a17..1836e35ac4 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -3479,7 +3479,7 @@ include_dir 'conf.d' Maximum size to let the WAL grow during automatic checkpoints. This is a soft limit; WAL size can exceed max_wal_size under special circumstances, such as - heavy load, a failing archive_command, or a high + heavy load, a failing archive_library, or a high wal_keep_size setting. If this value is specified without units, it is taken as megabytes. The default is 1 GB. @@ -3528,7 +3528,7 @@ include_dir 'conf.d' When archive_mode is enabled, completed WAL segments are sent to archive storage by setting - . In addition to off, + . In addition to off, to disable, there are two modes: on, and always. During normal operation, there is no difference between the two modes, but when set to always @@ -3538,9 +3538,6 @@ include_dir 'conf.d' for details. - archive_mode and archive_command are - separate variables so that archive_command can be - changed without leaving archiving mode. This parameter can only be set at server start. archive_mode cannot be enabled when wal_level is set to minimal. @@ -3548,6 +3545,28 @@ include_dir 'conf.d' + + archive_library (string) + + archive_library configuration parameter + + + + + The library to use for archiving completed WAL file segments. If set to + shell (the default) or an empty string, archiving via + shell is enabled, and is used. + Otherwise, the specified shared library is used for archiving. For more + information, see and + . + + + This parameter can only be set in the + postgresql.conf file or on the server command line. + + + + archive_command (string) @@ -3570,9 +3589,11 @@ include_dir 'conf.d' This parameter can only be set in the postgresql.conf file or on the server command line. It is ignored unless - archive_mode was enabled at server start. + archive_mode was enabled at server start and + archive_library specifies to archive via shell command. If archive_command is an empty string (the default) while - archive_mode is enabled, WAL archiving is temporarily + archive_mode is enabled and archive_library + specifies archiving via shell, WAL archiving is temporarily disabled, but the server continues to accumulate WAL segment files in the expectation that a command will soon be provided. Setting archive_command to a command that does nothing but @@ -3592,7 +3613,7 @@ include_dir 'conf.d' - The is only invoked for + The is only invoked for completed WAL segments. Hence, if your server generates little WAL traffic (or has slack periods where it does so), there could be a long delay between the completion of a transaction and its safe diff --git a/doc/src/sgml/contrib.sgml b/doc/src/sgml/contrib.sgml index d3ca4b6932..be9711c6f2 100644 --- a/doc/src/sgml/contrib.sgml +++ b/doc/src/sgml/contrib.sgml @@ -99,6 +99,7 @@ CREATE EXTENSION module_name; &amcheck; &auth-delay; &auto-explain; + &basic-archive; &bloom; &btree-gin; &btree-gist; diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml index 89454e99b9..328cd1f378 100644 --- a/doc/src/sgml/filelist.sgml +++ b/doc/src/sgml/filelist.sgml @@ -99,6 +99,7 @@ + @@ -112,6 +113,7 @@ + diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml index a265409f02..437712762a 100644 --- a/doc/src/sgml/high-availability.sgml +++ b/doc/src/sgml/high-availability.sgml @@ -935,7 +935,7 @@ primary_conninfo = 'host=192.168.1.50 port=5432 user=foo password=foopass' In lieu of using replication slots, it is possible to prevent the removal of old WAL segments using , or by storing the segments in an archive using - . + . However, these methods often result in retaining more WAL segments than required, whereas replication slots retain only the number of segments known to be needed. On the other hand, replication slots can retain so @@ -1386,10 +1386,10 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)' to always, and the standby will call the archive command for every WAL segment it receives, whether it's by restoring from the archive or by streaming replication. The shared archive can - be handled similarly, but the archive_command must + be handled similarly, but the archive_library must test if the file being archived exists already, and if the existing file has identical contents. This requires more care in the - archive_command, as it must + archive_library, as it must be careful to not overwrite an existing file with different contents, but return success if the exactly same file is archived twice. And all that must be done free of race conditions, if two servers attempt diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml index dba9cf413f..3db6d2160b 100644 --- a/doc/src/sgml/postgres.sgml +++ b/doc/src/sgml/postgres.sgml @@ -233,6 +233,7 @@ break is not needed in a wider output rendering. &bgworker; &logicaldecoding; &replication-origins; + &archive-modules; diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml index 1546f10c0d..e7ae29ec3d 100644 --- a/doc/src/sgml/ref/pg_basebackup.sgml +++ b/doc/src/sgml/ref/pg_basebackup.sgml @@ -102,8 +102,8 @@ PostgreSQL documentation All WAL records required for the backup must contain sufficient full-page writes, which requires you to enable full_page_writes on the primary and - not to use a tool like pg_compresslog as - archive_command to remove full-page writes from WAL files. + not to use a tool in your archive_library to remove + full-page writes from WAL files. diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml index b2e41ea814..b846213fb7 100644 --- a/doc/src/sgml/ref/pg_receivewal.sgml +++ b/doc/src/sgml/ref/pg_receivewal.sgml @@ -40,7 +40,7 @@ PostgreSQL documentation pg_receivewal streams the write-ahead log in real time as it's being generated on the server, and does not wait - for segments to complete like does. + for segments to complete like does. For this reason, it is not necessary to set when using pg_receivewal. @@ -487,11 +487,11 @@ PostgreSQL documentation When using pg_receivewal instead of - as the main WAL backup method, it is + as the main WAL backup method, it is strongly recommended to use replication slots. Otherwise, the server is free to recycle or remove write-ahead log files before they are backed up, because it does not have any information, either - from or the replication slots, about + from or the replication slots, about how far the WAL stream has been archived. Note, however, that a replication slot will fill up the server's disk space if the receiver does not keep up with fetching the WAL data. diff --git a/doc/src/sgml/wal.sgml b/doc/src/sgml/wal.sgml index 24e1c89503..2bb27a8468 100644 --- a/doc/src/sgml/wal.sgml +++ b/doc/src/sgml/wal.sgml @@ -636,7 +636,7 @@ WAL files plus one additional WAL file are kept at all times. Also, if WAL archiving is used, old segments cannot be removed or recycled until they are archived. If WAL archiving cannot keep up - with the pace that WAL is generated, or if archive_command + with the pace that WAL is generated, or if archive_library fails repeatedly, old WAL files will accumulate in pg_wal until the situation is resolved. A slow or failed standby server that uses a replication slot will have the same effect (see -- 2.25.1 --zhXaljGHf11kAtnf--