pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedTwo pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
46+ messages / 8 participants
[nested] [flat]
Commits from this thread:
2019-10-08 55ba56415b Improve test coverage of pg_rewind
2019-10-07 caa078353e Improve handling and coverage of --no-ensure-shutdown in pg_rewind
2019-10-04 6837632b75 Fix issues in pg_rewind with --no-ensure-shutdown/--write-recovery-conf
2019-10-04 6f3823b035 c2e3b311d9 Fix --dry-run mode of pg_rewind
2019-09-30 7524c78874 pg_rewind: test new --write-recovery-conf functionality
2019-09-30 927474ce1a pg_rewind: Allow writing recovery configuration
2019-09-27 5adafaf176 Have pg_rewind run crash recovery before rewinding
2019-09-25 caba97a9d9 Split out recovery confing-writing code from pg_basebackup
* Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
@ 2019-03-19 06:09 Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-09-30 17:15 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 2 replies; 46+ messages in thread
From: Paul Guo @ 2019-03-19 06:09 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
Hello, Postgres hackers,
Please see the attached patches.
The first patch adds an option to automatically generate recovery conf
contents in related files, following pg_basebackup. In the patch,
GenerateRecoveryConf(), WriteRecoveryConf() and escape_quotes() are almost
same as them on pg_basebackup. The main difference is due to replication
slot support and code (variables) limit. It seems that we could slightly
refactor later to put some common code into another file after aligning
pg_rewind with pg_basebackup. This was tested manually and was done by
Jimmy (cc-ed), Ashiwin (cc-ed) and me.
Another patch does automatic clean shutdown by running a single mode
postgres instance if the target was not clean shut down since that is
required by pg_rewind. This was manually tested and was done by Jimmy
(cc-ed) and me. I'm not sure if we want a test case for that though.
Thanks.
Attachments: download all (2 files, .tar.gz)
0001-Ensure-target-clean-shutdown-at-beginning-of-pg_rewi.patch (application/octet-stream, 3.0K) download
From 50b2192e0c7f64d63ab3571b62e563140485dea9 Mon Sep 17 00:00:00 2001
From: Paul Guo <pguo@pivotal.io>
Date: Tue, 19 Mar 2019 12:41:30 +0800
Subject: [PATCH] Ensure target clean shutdown at beginning of pg_rewind
---
src/bin/pg_rewind/pg_rewind.c | 60 +++++++++++++++++++++++++++++++++++
1 file changed, 60 insertions(+)
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index aa753bb315..946edec62c 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -41,6 +41,7 @@ static void updateControlFile(ControlFileData *ControlFile);
static void syncTargetDirectory(void);
static void sanityChecks(void);
static void findCommonAncestorTimeline(XLogRecPtr *recptr, int *tliIndex);
+static void ensureCleanShutdown(const char *argv0);
static ControlFileData ControlFile_target;
static ControlFileData ControlFile_source;
@@ -235,6 +236,21 @@ main(int argc, char **argv)
digestControlFile(&ControlFile_target, buffer, size);
pg_free(buffer);
+ /*
+ * If the target instance was not cleanly shut down, run a single-user
+ * postgres session really quickly and reload the control file to get the
+ * new state.
+ */
+ if (ControlFile_target.state != DB_SHUTDOWNED &&
+ ControlFile_target.state != DB_SHUTDOWNED_IN_RECOVERY)
+ {
+ ensureCleanShutdown(argv[0]);
+
+ buffer = slurpFile(datadir_target, "global/pg_control", &size);
+ digestControlFile(&ControlFile_target, buffer, size);
+ pg_free(buffer);
+ }
+
buffer = fetchFile("global/pg_control", &size);
digestControlFile(&ControlFile_source, buffer, size);
pg_free(buffer);
@@ -722,3 +738,47 @@ syncTargetDirectory(void)
fsync_pgdata(datadir_target, progname, PG_VERSION_NUM);
}
+
+/*
+ * Ensure clean shutdown of target instance by launching single-user mode
+ * postgres to do crash recovery.
+ */
+static void
+ensureCleanShutdown(const char *argv0)
+{
+ int ret;
+#define MAXCMDLEN (2 * MAXPGPATH)
+ char exec_path[MAXPGPATH];
+ char cmd[MAXCMDLEN];
+
+ /* locate postgres binary */
+ if ((ret = find_other_exec(argv0, "postgres",
+ PG_BACKEND_VERSIONSTR,
+ exec_path)) < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(argv0, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+
+ if (ret == -1)
+ pg_fatal("The program \"postgres\" is needed by %s but was \n"
+ "not found in the same directory as \"%s\".\n"
+ "Check your installation.\n", progname, full_path);
+ else
+ pg_fatal("The program \"postgres\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.\n", full_path, progname);
+ }
+
+ /* only skip processing after ensuring presence of postgres */
+ if (dry_run)
+ return;
+
+ /* finally run postgres single-user mode */
+ snprintf(cmd, MAXCMDLEN, "\"%s\" --single -D \"%s\" template1 < %s",
+ exec_path, datadir_target, DEVNULL);
+
+ if (system(cmd) != 0)
+ pg_fatal("postgres single-user mode of target instance failed for command: %s\n", cmd);
+}
--
2.17.2
0001-Auto-generate-recovery-conf-at-the-end-of-pg_re.patch (application/octet-stream, 8.4K) download
From abb1068b10a5f558623d6fc8f214c29d09f6bb28 Mon Sep 17 00:00:00 2001
From: Jimmy Yih <jyih@pivotal.io>
Date: Sun, 4 Nov 2018 22:54:53 -0800
Subject: [PATCH] Auto generate recovery conf at the end of pg_rewind
---
src/bin/pg_rewind/Makefile | 2 +-
src/bin/pg_rewind/fetch.h | 3 +
src/bin/pg_rewind/libpq_fetch.c | 138 ++++++++++++++++++++++++++++++++
src/bin/pg_rewind/pg_rewind.c | 23 +++++-
src/bin/pg_rewind/pg_rewind.h | 2 +
6 files changed, 166 insertions(+), 3 deletions(-)
diff --git a/src/bin/pg_rewind/Makefile b/src/bin/pg_rewind/Makefile
index 04f3b8f520..3b7f1ce177 100644
--- a/src/bin/pg_rewind/Makefile
+++ b/src/bin/pg_rewind/Makefile
@@ -16,7 +16,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I$(libpq_srcdir) -DFRONTEND $(CPPFLAGS)
-LDFLAGS_INTERNAL += $(libpq_pgport)
+LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
OBJS = pg_rewind.o parsexlog.o xlogreader.o datapagemap.o timeline.o \
fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o logging.o \
diff --git a/src/bin/pg_rewind/fetch.h b/src/bin/pg_rewind/fetch.h
index a694e8b157..7db9337ced 100644
--- a/src/bin/pg_rewind/fetch.h
+++ b/src/bin/pg_rewind/fetch.h
@@ -41,4 +41,7 @@ extern void copy_executeFileMap(filemap_t *map);
typedef void (*process_file_callback_t) (const char *path, file_type_t type, size_t size, const char *link_target);
extern void traverse_datadir(const char *datadir, process_file_callback_t callback);
+extern void GenerateRecoveryConf(void);
+extern void WriteRecoveryConf(void);
+
#endif /* FETCH_H */
diff --git a/src/bin/pg_rewind/libpq_fetch.c b/src/bin/pg_rewind/libpq_fetch.c
index d06e277432..47a3f68c93 100644
--- a/src/bin/pg_rewind/libpq_fetch.c
+++ b/src/bin/pg_rewind/libpq_fetch.c
@@ -24,10 +24,15 @@
#include "libpq-fe.h"
#include "catalog/pg_type_d.h"
#include "fe_utils/connect.h"
+#include "fe_utils/string_utils.h"
#include "port/pg_bswap.h"
+#include "pqexpbuffer.h"
static PGconn *conn = NULL;
+/* Contents of configuration to be generated */
+static PQExpBuffer recoveryconfcontents = NULL;
+
/*
* Files are fetched max CHUNKSIZE bytes at a time.
*
@@ -526,3 +531,136 @@ execute_pagemap(datapagemap_t *pagemap, const char *path)
}
pg_free(iter);
}
+
+/*
+ * Escape a string so that it can be used as a value in a key-value pair
+ * a configuration file.
+ */
+static char *
+escape_quotes(const char *src)
+{
+ char *result = escape_single_quotes_ascii(src);
+
+ if (!result)
+ {
+ fprintf(stderr, _("%s: out of memory\n"), progname);
+ exit(1);
+ }
+ return result;
+}
+
+/*
+ * Create a configuration file in memory using a PQExpBuffer
+ */
+void
+GenerateRecoveryConf(void)
+{
+ PQconninfoOption *connOptions;
+ PQconninfoOption *option;
+ PQExpBufferData conninfo_buf;
+ char *escaped;
+
+ recoveryconfcontents = createPQExpBuffer();
+ if (!recoveryconfcontents)
+ {
+ fprintf(stderr, _("%s: out of memory\n"), progname);
+ exit(1);
+ }
+
+ connOptions = PQconninfo(conn);
+ if (connOptions == NULL)
+ {
+ fprintf(stderr, _("%s: out of memory\n"), progname);
+ exit(1);
+ }
+
+ initPQExpBuffer(&conninfo_buf);
+ for (option = connOptions; option && option->keyword; option++)
+ {
+ /*
+ * Do not emit this setting if: - the setting is "replication",
+ * "dbname" or "fallback_application_name", since these would be
+ * overridden by the libpqwalreceiver module anyway. - not set or
+ * empty.
+ */
+ if (strcmp(option->keyword, "replication") == 0 ||
+ strcmp(option->keyword, "dbname") == 0 ||
+ strcmp(option->keyword, "fallback_application_name") == 0 ||
+ (option->val == NULL) ||
+ (option->val != NULL && option->val[0] == '\0'))
+ continue;
+
+ /* Separate key-value pairs with spaces */
+ if (conninfo_buf.len != 0)
+ appendPQExpBufferChar(&conninfo_buf, ' ');
+
+ /*
+ * Write "keyword=value" pieces, the value string is escaped and/or
+ * quoted if necessary.
+ */
+ appendPQExpBuffer(&conninfo_buf, "%s=", option->keyword);
+ appendConnStrVal(&conninfo_buf, option->val);
+ }
+
+ /*
+ * Escape the connection string, so that it can be put in the config file.
+ * Note that this is different from the escaping of individual connection
+ * options above!
+ */
+ escaped = escape_quotes(conninfo_buf.data);
+ appendPQExpBuffer(recoveryconfcontents, "primary_conninfo = '%s'\n", escaped);
+ free(escaped);
+
+ if (PQExpBufferBroken(recoveryconfcontents) ||
+ PQExpBufferDataBroken(conninfo_buf))
+ {
+ fprintf(stderr, _("%s: out of memory\n"), progname);
+ exit(1);
+ }
+
+ termPQExpBuffer(&conninfo_buf);
+
+ PQconninfoFree(connOptions);
+}
+
+
+/*
+ * Write the configuration file into the directory specified in datadir_target,
+ * with the contents already collected in memory.
+ * Then write the signal file into the datadir_target also.
+ */
+void
+WriteRecoveryConf(void)
+{
+ char filename[MAXPGPATH];
+ FILE *cf;
+
+ snprintf(filename, MAXPGPATH, "%s/%s", datadir_target, "postgresql.auto.conf");
+
+ cf = fopen(filename, "a");
+ if (cf == NULL)
+ {
+ fprintf(stderr, _("%s: could not open file \"%s\": %s\n"), progname, filename, strerror(errno));
+ exit(1);
+ }
+
+ if (fwrite(recoveryconfcontents->data, recoveryconfcontents->len, 1, cf) != 1)
+ {
+ fprintf(stderr,
+ _("%s: could not write to file \"%s\": %s\n"),
+ progname, filename, strerror(errno));
+ exit(1);
+ }
+
+ fclose(cf);
+
+ snprintf(filename, MAXPGPATH, "%s/%s", datadir_target, "standby.signal");
+ cf = fopen(filename, "w");
+ if (cf == NULL)
+ {
+ fprintf(stderr, _("%s: could not create file \"%s\": %s\n"), progname, filename, strerror(errno));
+ exit(1);
+ }
+
+ fclose(cf);
+}
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index aa753bb315..b6b288bbfe 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -45,6 +45,8 @@ static void findCommonAncestorTimeline(XLogRecPtr *recptr, int *tliIndex);
static ControlFileData ControlFile_target;
static ControlFileData ControlFile_source;
+static bool writerecoveryconf = false;
+
const char *progname;
int WalSegSz;
@@ -71,6 +73,7 @@ usage(const char *progname)
printf(_(" -D, --target-pgdata=DIRECTORY existing data directory to modify\n"));
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
+ printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"));
printf(_(" safely to disk\n"));
@@ -88,6 +91,7 @@ main(int argc, char **argv)
static struct option long_options[] = {
{"help", no_argument, NULL, '?'},
{"target-pgdata", required_argument, NULL, 'D'},
+ {"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
{"version", no_argument, NULL, 'V'},
@@ -129,7 +133,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNP", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -149,6 +153,10 @@ main(int argc, char **argv)
do_sync = false;
break;
+ case 'R':
+ writerecoveryconf = true;
+ break;
+
case 3:
debug = true;
break;
@@ -291,6 +299,13 @@ main(int argc, char **argv)
if (!rewind_needed)
{
printf(_("no rewind required\n"));
+
+ if (writerecoveryconf && connstr_source)
+ {
+ GenerateRecoveryConf();
+ WriteRecoveryConf();
+ }
+
exit(0);
}
@@ -382,6 +397,12 @@ main(int argc, char **argv)
pg_log(PG_PROGRESS, "syncing target data directory\n");
syncTargetDirectory();
+ if (writerecoveryconf && connstr_source)
+ {
+ GenerateRecoveryConf();
+ WriteRecoveryConf();
+ }
+
printf(_("Done!\n"));
return 0;
diff --git a/src/bin/pg_rewind/pg_rewind.h b/src/bin/pg_rewind/pg_rewind.h
index 83b2898b8b..0ce9ba0426 100644
--- a/src/bin/pg_rewind/pg_rewind.h
+++ b/src/bin/pg_rewind/pg_rewind.h
@@ -30,6 +30,8 @@ extern int WalSegSz;
extern TimeLineHistoryEntry *targetHistory;
extern int targetNentries;
+extern const char *progname;
+
/* in parsexlog.c */
extern void extractPageMap(const char *datadir, XLogRecPtr startpoint,
int tliIndex, XLogRecPtr endpoint);
--
2.17.2
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-03-19 06:18 ` Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
1 sibling, 1 reply; 46+ messages in thread
From: Michael Paquier @ 2019-03-19 06:18 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: pgsql-hackers@postgresql.org, Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On Tue, Mar 19, 2019 at 02:09:03PM +0800, Paul Guo wrote:
> The first patch adds an option to automatically generate recovery conf
> contents in related files, following pg_basebackup. In the patch,
> GenerateRecoveryConf(), WriteRecoveryConf() and escape_quotes() are almost
> same as them on pg_basebackup. The main difference is due to replication
> slot support and code (variables) limit. It seems that we could slightly
> refactor later to put some common code into another file after aligning
> pg_rewind with pg_basebackup. This was tested manually and was done by
> Jimmy (cc-ed), Ashiwin (cc-ed) and me.
Interesting. The two routines have really the same logic, I would
recommend to have a first patch which does the refactoring and have
pg_rewind use it, and then a second patch which writes recovery.conf
and uses the first patch to get the contents. Please note that the
common routine needs to be version-aware as pg_basebackup requires
compatibility with past versions, but you could just pass the version
number from the connection, and have pg_rewind pass the compiled-in
version value.
> Another patch does automatic clean shutdown by running a single mode
> postgres instance if the target was not clean shut down since that is
> required by pg_rewind. This was manually tested and was done by Jimmy
> (cc-ed) and me. I'm not sure if we want a test case for that though.
I am not sure that I see the value in that. I'd rather let the
required service start and stop out of pg_rewind and not introduce
dependencies with other binaries. This step can also take quite some
time depending on the amount of WAL to replay post-crash at recovery
and the shutdown checkpoint which is required to reach a consistent
on-disk state.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
@ 2019-03-20 04:48 ` Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 46+ messages in thread
From: Paul Guo @ 2019-03-20 04:48 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers@postgresql.org, Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On Tue, Mar 19, 2019 at 2:18 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Tue, Mar 19, 2019 at 02:09:03PM +0800, Paul Guo wrote:
> > The first patch adds an option to automatically generate recovery conf
> > contents in related files, following pg_basebackup. In the patch,
> > GenerateRecoveryConf(), WriteRecoveryConf() and escape_quotes() are
> almost
> > same as them on pg_basebackup. The main difference is due to replication
> > slot support and code (variables) limit. It seems that we could slightly
> > refactor later to put some common code into another file after aligning
> > pg_rewind with pg_basebackup. This was tested manually and was done by
> > Jimmy (cc-ed), Ashiwin (cc-ed) and me.
>
>
> Interesting. The two routines have really the same logic, I would
> recommend to have a first patch which does the refactoring and have
> pg_rewind use it, and then a second patch which writes recovery.conf
> and uses the first patch to get the contents. Please note that the
>
This is a good suggestion also. Will do it.
> common routine needs to be version-aware as pg_basebackup requires
> compatibility with past versions, but you could just pass the version
> number from the connection, and have pg_rewind pass the compiled-in
> version value.
>
> > Another patch does automatic clean shutdown by running a single mode
> > postgres instance if the target was not clean shut down since that is
> > required by pg_rewind. This was manually tested and was done by Jimmy
> > (cc-ed) and me. I'm not sure if we want a test case for that though.
>
> I am not sure that I see the value in that. I'd rather let the
> required service start and stop out of pg_rewind and not introduce
> dependencies with other binaries. This step can also take quite some
>
This makes recovery more automatically. Yes, it will add the dependency on
the postgres
binary, but it seems that most time pg_rewind should be shipped as postgres
in the same install directory. From my experience of manually testing
pg_rewind,
I feel that this besides auto-recovery-conf writing really alleviate my
burden. I'm not sure how
other users usually do before running pg_rewind when the target is not
cleanly shut down,
but probably we can add an argument to pg_rewind to give those people who
want to
handle target separately another chance? default on or off whatever.
> time depending on the amount of WAL to replay post-crash at recovery
> and the shutdown checkpoint which is required to reach a consistent
> on-disk state.
>
The time is still required for people who want to make the target ready for
pg_rewind in another way.
Thanks.
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-03-20 05:20 ` Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
0 siblings, 1 reply; 46+ messages in thread
From: Michael Paquier @ 2019-03-20 05:20 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: pgsql-hackers@postgresql.org, Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On Wed, Mar 20, 2019 at 12:48:52PM +0800, Paul Guo wrote:
> This is a good suggestion also. Will do it.
Please note also that we don't care about recovery.conf since v12 as
recovery parameters are now GUCs. I would suggest appending those
extra parameters to postgresql.auto.conf, which is what pg_basebackup
does.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
@ 2019-03-20 05:23 ` Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
0 siblings, 1 reply; 46+ messages in thread
From: Paul Guo @ 2019-03-20 05:23 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers@postgresql.org, Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On Wed, Mar 20, 2019 at 1:20 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Wed, Mar 20, 2019 at 12:48:52PM +0800, Paul Guo wrote:
> > This is a good suggestion also. Will do it.
>
> Please note also that we don't care about recovery.conf since v12 as
> recovery parameters are now GUCs. I would suggest appending those
> extra parameters to postgresql.auto.conf, which is what pg_basebackup
> does.
>
Yes, the recovery conf patch in the first email did like this, i.e. writing
postgresql.auto.conf & standby.signal
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-04-19 03:40 ` Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-01 15:57 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 2 replies; 46+ messages in thread
From: Paul Guo @ 2019-04-19 03:40 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers@postgresql.org, Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
Hi Michael,
I updated the patches as attached following previous discussions.
The two patches:
v2-0001-Extact-common-functions-from-pg_basebackup-into-s.patch
v2-0002-Add-option-to-write-recovery-configuration-inform.patch
1. 0001 does move those common functions & variables to two new files (one
.c and one .h) for both pg_rewind and pg_basebackup use,
note the functions are slightly modified (e.g. because conn is probably
NULL on pg_rewind). I do not know where is more proper to put the
new files. Currently, they are under pg_basebackup and are used in
pg_rewind (Makefile modified to support that).
2. 0002 adds the option to write recovery conf.
The below patch runs single mode Postgres if needed to make sure the target
is cleanly shutdown. A new option is added (off by default).
v2-0001-Ensure-target-clean-shutdown-at-beginning-of-pg_r.patch
I've manually tested them and installcheck passes.
Thanks.
On Wed, Mar 20, 2019 at 1:23 PM Paul Guo <pguo@pivotal.io> wrote:
>
>
> On Wed, Mar 20, 2019 at 1:20 PM Michael Paquier <michael@paquier.xyz>
> wrote:
>
>> On Wed, Mar 20, 2019 at 12:48:52PM +0800, Paul Guo wrote:
>> > This is a good suggestion also. Will do it.
>>
>> Please note also that we don't care about recovery.conf since v12 as
>> recovery parameters are now GUCs. I would suggest appending those
>> extra parameters to postgresql.auto.conf, which is what pg_basebackup
>> does.
>>
> Yes, the recovery conf patch in the first email did like this, i.e.
> writing postgresql.auto.conf & standby.signal
>
>
Attachments: download all (3 files, .tar.gz)
v2-0001-Ensure-target-clean-shutdown-at-beginning-of-pg_r.patch (application/octet-stream, 4.5K) download
From 3afa833cf414187bf3c34e092666ad9117ab38ae Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Tue, 19 Mar 2019 12:41:30 +0800
Subject: [PATCH v2] Ensure target clean shutdown at beginning of pg_rewind
---
src/bin/pg_rewind/pg_rewind.c | 68 +++++++++++++++++++++++++++++++++++
1 file changed, 68 insertions(+)
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index b7e6aca97f..36001deada 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -42,6 +42,7 @@ static void digestControlFile(ControlFileData *ControlFile, char *source,
static void syncTargetDirectory(void);
static void sanityChecks(void);
static void findCommonAncestorTimeline(XLogRecPtr *recptr, int *tliIndex);
+static void ensureCleanShutdown(const char *argv0);
static ControlFileData ControlFile_target;
static ControlFileData ControlFile_source;
@@ -72,6 +73,7 @@ usage(const char *progname)
printf(_(" -D, --target-pgdata=DIRECTORY existing data directory to modify\n"));
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
+ printf(_(" -e, --ensure-clean-shutdown run single-mode postgres if needed to make sure target is clean shutdown\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"));
printf(_(" safely to disk\n"));
@@ -91,6 +93,7 @@ main(int argc, char **argv)
{"target-pgdata", required_argument, NULL, 'D'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
+ {"ensure-clean-shutdown", no_argument, NULL, 'e'},
{"version", no_argument, NULL, 'V'},
{"dry-run", no_argument, NULL, 'n'},
{"no-sync", no_argument, NULL, 'N'},
@@ -107,6 +110,7 @@ main(int argc, char **argv)
XLogRecPtr chkptredo;
size_t size;
char *buffer;
+ bool ensure_clean_shutdown = false;
bool rewind_needed;
XLogRecPtr endrec;
TimeLineID endtli;
@@ -139,6 +143,10 @@ main(int argc, char **argv)
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
+ case 'e':
+ ensure_clean_shutdown = true;
+ break;
+
case 'P':
showprogress = true;
break;
@@ -238,6 +246,22 @@ main(int argc, char **argv)
digestControlFile(&ControlFile_target, buffer, size);
pg_free(buffer);
+ /*
+ * If the target instance was not cleanly shut down, run a single-user
+ * postgres session really quickly and reload the control file to get the
+ * new state.
+ */
+ if (ensure_clean_shutdown &&
+ ControlFile_target.state != DB_SHUTDOWNED &&
+ ControlFile_target.state != DB_SHUTDOWNED_IN_RECOVERY)
+ {
+ ensureCleanShutdown(argv[0]);
+
+ buffer = slurpFile(datadir_target, "global/pg_control", &size);
+ digestControlFile(&ControlFile_target, buffer, size);
+ pg_free(buffer);
+ }
+
buffer = fetchFile("global/pg_control", &size);
digestControlFile(&ControlFile_source, buffer, size);
pg_free(buffer);
@@ -690,3 +714,47 @@ syncTargetDirectory(void)
fsync_pgdata(datadir_target, PG_VERSION_NUM);
}
+
+/*
+ * Ensure clean shutdown of target instance by launching single-user mode
+ * postgres to do crash recovery.
+ */
+static void
+ensureCleanShutdown(const char *argv0)
+{
+ int ret;
+#define MAXCMDLEN (2 * MAXPGPATH)
+ char exec_path[MAXPGPATH];
+ char cmd[MAXCMDLEN];
+
+ /* locate postgres binary */
+ if ((ret = find_other_exec(argv0, "postgres",
+ PG_BACKEND_VERSIONSTR,
+ exec_path)) < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(argv0, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+
+ if (ret == -1)
+ pg_fatal("The program \"postgres\" is needed by %s but was \n"
+ "not found in the same directory as \"%s\".\n"
+ "Check your installation.\n", progname, full_path);
+ else
+ pg_fatal("The program \"postgres\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.\n", full_path, progname);
+ }
+
+ /* only skip processing after ensuring presence of postgres */
+ if (dry_run)
+ return;
+
+ /* finally run postgres single-user mode */
+ snprintf(cmd, MAXCMDLEN, "\"%s\" --single -D \"%s\" template1 < %s",
+ exec_path, datadir_target, DEVNULL);
+
+ if (system(cmd) != 0)
+ pg_fatal("postgres single-user mode of target instance failed for command: %s\n", cmd);
+}
--
2.17.2
v2-0001-Extact-common-functions-from-pg_basebackup-into-s.patch (application/octet-stream, 17.1K) download
From 865827daa52d69fed40e48171e0b4d33383f51fa Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 18:55:09 +0800
Subject: [PATCH v2 1/2] Extact common functions from pg_basebackup into
separate files for pg_rewind use.
---
src/bin/pg_basebackup/Makefile | 2 +-
src/bin/pg_basebackup/backup_common.c | 181 ++++++++++++++++++++++++++
src/bin/pg_basebackup/backup_common.h | 33 +++++
src/bin/pg_basebackup/pg_basebackup.c | 179 +------------------------
src/bin/pg_basebackup/streamutil.c | 1 -
src/bin/pg_rewind/Makefile | 14 +-
src/bin/pg_rewind/libpq_fetch.c | 2 +-
src/bin/pg_rewind/pg_rewind.c | 4 +
8 files changed, 234 insertions(+), 182 deletions(-)
create mode 100644 src/bin/pg_basebackup/backup_common.c
create mode 100644 src/bin/pg_basebackup/backup_common.h
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index d7a081b9bb..2ed77bd0d6 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -21,7 +21,7 @@ include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
-OBJS=receivelog.o streamutil.o walmethods.o $(WIN32RES)
+OBJS=receivelog.o streamutil.o walmethods.o backup_common.o $(WIN32RES)
all: pg_basebackup pg_receivewal pg_recvlogical
diff --git a/src/bin/pg_basebackup/backup_common.c b/src/bin/pg_basebackup/backup_common.c
new file mode 100644
index 0000000000..7a2accea12
--- /dev/null
+++ b/src/bin/pg_basebackup/backup_common.c
@@ -0,0 +1,181 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_common.c - common code that is used in both pg_basebackup and pg_rewind.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+#include "fe_utils/logging.h"
+#include "fe_utils/string_utils.h"
+#include "backup_common.h"
+
+/* Contents of configuration file to be generated */
+PQExpBuffer recoveryconfcontents = NULL;
+
+bool writerecoveryconf = false;
+char *replication_slot = NULL;
+PGconn *conn = NULL;
+
+void
+disconnect_atexit(void)
+{
+ if (conn != NULL)
+ PQfinish(conn);
+}
+
+/*
+ * Escape a string so that it can be used as a value in a key-value pair
+ * a configuration file.
+ */
+static char *
+escape_quotes(const char *src)
+{
+ char *result = escape_single_quotes_ascii(src);
+
+ if (!result)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+ return result;
+}
+
+/*
+ * Create a configuration file in memory using a PQExpBuffer
+ */
+void
+GenerateRecoveryConf(void)
+{
+ PQconninfoOption *connOptions;
+ PQconninfoOption *option;
+ PQExpBufferData conninfo_buf;
+ char *escaped;
+
+ recoveryconfcontents = createPQExpBuffer();
+ if (!recoveryconfcontents)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ /*
+ * In PostgreSQL 12 and newer versions, standby_mode is gone, replaced by
+ * standby.signal to trigger a standby state at recovery.
+ */
+ if (conn && PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
+ appendPQExpBufferStr(recoveryconfcontents, "standby_mode = 'on'\n");
+
+ connOptions = PQconninfo(conn);
+ if (connOptions == NULL)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ initPQExpBuffer(&conninfo_buf);
+ for (option = connOptions; option && option->keyword; option++)
+ {
+ /*
+ * Do not emit this setting if: - the setting is "replication",
+ * "dbname" or "fallback_application_name", since these would be
+ * overridden by the libpqwalreceiver module anyway. - not set or
+ * empty.
+ */
+ if (strcmp(option->keyword, "replication") == 0 ||
+ strcmp(option->keyword, "dbname") == 0 ||
+ strcmp(option->keyword, "fallback_application_name") == 0 ||
+ (option->val == NULL) ||
+ (option->val != NULL && option->val[0] == '\0'))
+ continue;
+
+ /* Separate key-value pairs with spaces */
+ if (conninfo_buf.len != 0)
+ appendPQExpBufferChar(&conninfo_buf, ' ');
+
+ /*
+ * Write "keyword=value" pieces, the value string is escaped and/or
+ * quoted if necessary.
+ */
+ appendPQExpBuffer(&conninfo_buf, "%s=", option->keyword);
+ appendConnStrVal(&conninfo_buf, option->val);
+ }
+
+ /*
+ * Escape the connection string, so that it can be put in the config file.
+ * Note that this is different from the escaping of individual connection
+ * options above!
+ */
+ escaped = escape_quotes(conninfo_buf.data);
+ appendPQExpBuffer(recoveryconfcontents, "primary_conninfo = '%s'\n", escaped);
+ free(escaped);
+
+ if (replication_slot)
+ {
+ escaped = escape_quotes(replication_slot);
+ appendPQExpBuffer(recoveryconfcontents, "primary_slot_name = '%s'\n", replication_slot);
+ free(escaped);
+ }
+
+ if (PQExpBufferBroken(recoveryconfcontents) ||
+ PQExpBufferDataBroken(conninfo_buf))
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ termPQExpBuffer(&conninfo_buf);
+
+ PQconninfoFree(connOptions);
+}
+
+/*
+ * Write the configuration file into the directory specified in basedir,
+ * with the contents already collected in memory appended. Then write
+ * the signal file into the basedir. If the server does not support
+ * recovery parameters as GUCs, the signal file is not necessary, and
+ * configuration is written to recovery.conf.
+ */
+void
+WriteRecoveryConf(char *target_dir)
+{
+ char filename[MAXPGPATH];
+ FILE *cf;
+ bool is_recovery_guc_supported = true;
+
+ if (conn && PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
+ is_recovery_guc_supported = false;
+
+ snprintf(filename, MAXPGPATH, "%s/%s", target_dir,
+ is_recovery_guc_supported ? "postgresql.auto.conf" : "recovery.conf");
+
+ cf = fopen(filename, is_recovery_guc_supported ? "a" : "w");
+ if (cf == NULL)
+ {
+ pg_log_error("could not open file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ if (fwrite(recoveryconfcontents->data, recoveryconfcontents->len, 1, cf) != 1)
+ {
+ pg_log_error("could not write to file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ fclose(cf);
+
+ if (is_recovery_guc_supported)
+ {
+ snprintf(filename, MAXPGPATH, "%s/%s", target_dir, "standby.signal");
+ cf = fopen(filename, "w");
+ if (cf == NULL)
+ {
+ pg_log_error("could not create file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ fclose(cf);
+ }
+}
diff --git a/src/bin/pg_basebackup/backup_common.h b/src/bin/pg_basebackup/backup_common.h
new file mode 100644
index 0000000000..8f9f860485
--- /dev/null
+++ b/src/bin/pg_basebackup/backup_common.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_common.h - common code that is used in both pg_basebackup and pg_rewind.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef BACKUP_COMMON_H
+#define BACKUP_COMMON_H
+
+#include "libpq-fe.h"
+#include "pqexpbuffer.h"
+
+/* Contents of configuration file to be generated */
+extern PQExpBuffer recoveryconfcontents;
+
+extern bool writerecoveryconf;
+extern char *replication_slot;
+PGconn *conn;
+
+extern void disconnect_atexit(void);
+extern void GenerateRecoveryConf(void);
+extern void WriteRecoveryConf(char *target_dir);
+
+/*
+ * recovery.conf is integrated into postgresql.conf from version 12.
+ */
+#define MINIMUM_VERSION_FOR_RECOVERY_GUC 120000
+
+
+#endif /* BACKUP_COMMON_H */
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 1a735b8046..21653144bc 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -37,6 +37,7 @@
#include "pqexpbuffer.h"
#include "pgtar.h"
#include "pgtime.h"
+#include "backup_common.h"
#include "receivelog.h"
#include "replication/basebackup.h"
#include "streamutil.h"
@@ -67,11 +68,6 @@ typedef struct TablespaceList
*/
#define MINIMUM_VERSION_FOR_TEMP_SLOTS 100000
-/*
- * recovery.conf is integrated into postgresql.conf from version 12.
- */
-#define MINIMUM_VERSION_FOR_RECOVERY_GUC 120000
-
/*
* Different ways to include WAL
*/
@@ -95,12 +91,10 @@ static int verbose = 0;
static int compresslevel = 0;
static IncludeWal includewal = STREAM_WAL;
static bool fastcheckpoint = false;
-static bool writerecoveryconf = false;
static bool do_sync = true;
static int standby_message_timeout = 10 * 1000; /* 10 sec = default */
static pg_time_t last_progress_report = 0;
static int32 maxrate = 0; /* no limit by default */
-static char *replication_slot = NULL;
static bool temp_replication_slot = true;
static bool create_slot = false;
static bool no_slot = false;
@@ -137,9 +131,6 @@ static int has_xlogendptr = 0;
static volatile LONG has_xlogendptr = 0;
#endif
-/* Contents of configuration file to be generated */
-static PQExpBuffer recoveryconfcontents = NULL;
-
/* Function headers */
static void usage(void);
static void verify_dir_is_empty_or_create(char *dirname, bool *created, bool *found);
@@ -147,8 +138,6 @@ static void progress_report(int tablespacenum, const char *filename, bool force)
static void ReceiveTarFile(PGconn *conn, PGresult *res, int rownum);
static void ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum);
-static void GenerateRecoveryConf(PGconn *conn);
-static void WriteRecoveryConf(void);
static void BaseBackup(void);
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
@@ -205,13 +194,6 @@ cleanup_directories_atexit(void)
pg_log_info("changes to tablespace directories will not be undone");
}
-static void
-disconnect_atexit(void)
-{
- if (conn != NULL)
- PQfinish(conn);
-}
-
#ifndef WIN32
/*
* On windows, our background thread dies along with the process. But on
@@ -1621,7 +1603,7 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
PQfreemem(copybuf);
if (basetablespace && writerecoveryconf)
- WriteRecoveryConf();
+ WriteRecoveryConf(basedir);
/*
* No data is synced here, everything is done for all tablespaces at the
@@ -1629,161 +1611,6 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
*/
}
-/*
- * Escape a string so that it can be used as a value in a key-value pair
- * a configuration file.
- */
-static char *
-escape_quotes(const char *src)
-{
- char *result = escape_single_quotes_ascii(src);
-
- if (!result)
- {
- pg_log_error("out of memory");
- exit(1);
- }
- return result;
-}
-
-/*
- * Create a configuration file in memory using a PQExpBuffer
- */
-static void
-GenerateRecoveryConf(PGconn *conn)
-{
- PQconninfoOption *connOptions;
- PQconninfoOption *option;
- PQExpBufferData conninfo_buf;
- char *escaped;
-
- recoveryconfcontents = createPQExpBuffer();
- if (!recoveryconfcontents)
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- /*
- * In PostgreSQL 12 and newer versions, standby_mode is gone, replaced by
- * standby.signal to trigger a standby state at recovery.
- */
- if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
- appendPQExpBufferStr(recoveryconfcontents, "standby_mode = 'on'\n");
-
- connOptions = PQconninfo(conn);
- if (connOptions == NULL)
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- initPQExpBuffer(&conninfo_buf);
- for (option = connOptions; option && option->keyword; option++)
- {
- /*
- * Do not emit this setting if: - the setting is "replication",
- * "dbname" or "fallback_application_name", since these would be
- * overridden by the libpqwalreceiver module anyway. - not set or
- * empty.
- */
- if (strcmp(option->keyword, "replication") == 0 ||
- strcmp(option->keyword, "dbname") == 0 ||
- strcmp(option->keyword, "fallback_application_name") == 0 ||
- (option->val == NULL) ||
- (option->val != NULL && option->val[0] == '\0'))
- continue;
-
- /* Separate key-value pairs with spaces */
- if (conninfo_buf.len != 0)
- appendPQExpBufferChar(&conninfo_buf, ' ');
-
- /*
- * Write "keyword=value" pieces, the value string is escaped and/or
- * quoted if necessary.
- */
- appendPQExpBuffer(&conninfo_buf, "%s=", option->keyword);
- appendConnStrVal(&conninfo_buf, option->val);
- }
-
- /*
- * Escape the connection string, so that it can be put in the config file.
- * Note that this is different from the escaping of individual connection
- * options above!
- */
- escaped = escape_quotes(conninfo_buf.data);
- appendPQExpBuffer(recoveryconfcontents, "primary_conninfo = '%s'\n", escaped);
- free(escaped);
-
- if (replication_slot)
- {
- escaped = escape_quotes(replication_slot);
- appendPQExpBuffer(recoveryconfcontents, "primary_slot_name = '%s'\n", replication_slot);
- free(escaped);
- }
-
- if (PQExpBufferBroken(recoveryconfcontents) ||
- PQExpBufferDataBroken(conninfo_buf))
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- termPQExpBuffer(&conninfo_buf);
-
- PQconninfoFree(connOptions);
-}
-
-
-/*
- * Write the configuration file into the directory specified in basedir,
- * with the contents already collected in memory appended. Then write
- * the signal file into the basedir. If the server does not support
- * recovery parameters as GUCs, the signal file is not necessary, and
- * configuration is written to recovery.conf.
- */
-static void
-WriteRecoveryConf(void)
-{
- char filename[MAXPGPATH];
- FILE *cf;
- bool is_recovery_guc_supported = true;
-
- if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
- is_recovery_guc_supported = false;
-
- snprintf(filename, MAXPGPATH, "%s/%s", basedir,
- is_recovery_guc_supported ? "postgresql.auto.conf" : "recovery.conf");
-
- cf = fopen(filename, is_recovery_guc_supported ? "a" : "w");
- if (cf == NULL)
- {
- pg_log_error("could not open file \"%s\": %m", filename);
- exit(1);
- }
-
- if (fwrite(recoveryconfcontents->data, recoveryconfcontents->len, 1, cf) != 1)
- {
- pg_log_error("could not write to file \"%s\": %m", filename);
- exit(1);
- }
-
- fclose(cf);
-
- if (is_recovery_guc_supported)
- {
- snprintf(filename, MAXPGPATH, "%s/%s", basedir, "standby.signal");
- cf = fopen(filename, "w");
- if (cf == NULL)
- {
- pg_log_error("could not create file \"%s\": %m", filename);
- exit(1);
- }
-
- fclose(cf);
- }
-}
-
static void
BaseBackup(void)
@@ -1840,7 +1667,7 @@ BaseBackup(void)
* Build contents of configuration file if requested
*/
if (writerecoveryconf)
- GenerateRecoveryConf(conn);
+ GenerateRecoveryConf();
/*
* Run IDENTIFY_SYSTEM so we can get the timeline
diff --git a/src/bin/pg_basebackup/streamutil.c b/src/bin/pg_basebackup/streamutil.c
index ab2e55d695..528d06e358 100644
--- a/src/bin/pg_basebackup/streamutil.c
+++ b/src/bin/pg_basebackup/streamutil.c
@@ -53,7 +53,6 @@ char *dbname = NULL;
int dbgetpassword = 0; /* 0=auto, -1=never, 1=always */
static bool have_password = false;
static char password[100];
-PGconn *conn = NULL;
/*
* Connect to the server. Returns a valid PGconn pointer if connected,
diff --git a/src/bin/pg_rewind/Makefile b/src/bin/pg_rewind/Makefile
index 019e19986e..7793702b20 100644
--- a/src/bin/pg_rewind/Makefile
+++ b/src/bin/pg_rewind/Makefile
@@ -19,16 +19,24 @@ override CPPFLAGS := -I$(libpq_srcdir) -DFRONTEND $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
OBJS = pg_rewind.o parsexlog.o xlogreader.o datapagemap.o timeline.o \
- fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o logging.o \
+ fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o logging.o backup_common.o \
$(WIN32RES)
-EXTRA_CLEAN = xlogreader.c
+EXTRA_CLEAN = xlogreader.c backup_common.c backup_common.h
all: pg_rewind
pg_rewind: $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_rewind.o: backup_common.h
+
+backup_common.c: % : $(top_srcdir)/src/bin/pg_basebackup/%
+ rm -f $@ && $(LN_S) $< .
+
+backup_common.h: % : $(top_srcdir)/src/bin/pg_basebackup/%
+ rm -f $@ && $(LN_S) $< .
+
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
@@ -42,7 +50,7 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_rewind$(X)'
clean distclean maintainer-clean:
- rm -f pg_rewind$(X) $(OBJS) xlogreader.c
+ rm -f pg_rewind$(X) $(OBJS) xlogreader.c backup_common.c backup_common.h
rm -rf tmp_check
check:
diff --git a/src/bin/pg_rewind/libpq_fetch.c b/src/bin/pg_rewind/libpq_fetch.c
index 11ec045b8e..3c601bdf3c 100644
--- a/src/bin/pg_rewind/libpq_fetch.c
+++ b/src/bin/pg_rewind/libpq_fetch.c
@@ -27,7 +27,7 @@
#include "fe_utils/logging.h"
#include "port/pg_bswap.h"
-static PGconn *conn = NULL;
+extern PGconn *conn;
/*
* Files are fetched max CHUNKSIZE bytes at a time.
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index b7e6aca97f..7337bdd833 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -31,6 +31,7 @@
#include "fe_utils/logging.h"
#include "getopt_long.h"
#include "storage/bufpage.h"
+#include "backup_common.h"
static void usage(const char *progname);
@@ -228,7 +229,10 @@ main(int argc, char **argv)
/* Connect to remote server */
if (connstr_source)
+ {
libpqConnect(connstr_source);
+ atexit(disconnect_atexit);
+ }
/*
* Ok, we have all the options and we're ready to start. Read in all the
--
2.17.2
v2-0002-Add-option-to-write-recovery-configuration-inform.patch (application/octet-stream, 2.5K) download
From 8d947baf848f6f8d81155b9354e9ad11cd6a3c86 Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 19:37:36 +0800
Subject: [PATCH v2 2/2] Add option to write recovery configuration information
in pg_rewind.
---
src/bin/pg_rewind/pg_rewind.c | 19 ++++++++++++++++++-
1 file changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index 7337bdd833..65d5a5d26d 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -73,6 +73,7 @@ usage(const char *progname)
printf(_(" -D, --target-pgdata=DIRECTORY existing data directory to modify\n"));
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
+ printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"));
printf(_(" safely to disk\n"));
@@ -90,6 +91,7 @@ main(int argc, char **argv)
static struct option long_options[] = {
{"help", no_argument, NULL, '?'},
{"target-pgdata", required_argument, NULL, 'D'},
+ {"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
{"version", no_argument, NULL, 'V'},
@@ -132,7 +134,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNP", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -152,6 +154,10 @@ main(int argc, char **argv)
do_sync = false;
break;
+ case 'R':
+ writerecoveryconf = true;
+ break;
+
case 3:
debug = true;
pg_logging_set_level(PG_LOG_DEBUG);
@@ -298,6 +304,11 @@ main(int argc, char **argv)
if (!rewind_needed)
{
pg_log_info("no rewind required");
+ if (writerecoveryconf && connstr_source)
+ {
+ GenerateRecoveryConf();
+ WriteRecoveryConf(datadir_target);
+ }
exit(0);
}
@@ -395,6 +406,12 @@ main(int argc, char **argv)
pg_log_info("syncing target data directory");
syncTargetDirectory();
+ if (writerecoveryconf && connstr_source)
+ {
+ GenerateRecoveryConf();
+ WriteRecoveryConf(datadir_target);
+ }
+
pg_log_info("Done!");
return 0;
--
2.17.2
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-07-01 11:34 ` Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
1 sibling, 1 reply; 46+ messages in thread
From: Thomas Munro @ 2019-07-01 11:34 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On Fri, Apr 19, 2019 at 3:41 PM Paul Guo <pguo@pivotal.io> wrote:
> I updated the patches as attached following previous discussions.
Hi Paul,
Could we please have a fresh rebase now that the CF is here?
Thanks,
--
Thomas Munro
https://enterprisedb.com
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
@ 2019-07-02 05:46 ` Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 46+ messages in thread
From: Paul Guo @ 2019-07-02 05:46 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
Rebased, aligned with recent changes in pg_rewind/pg_basebackup and then
retested. Thanks.
On Mon, Jul 1, 2019 at 7:35 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Fri, Apr 19, 2019 at 3:41 PM Paul Guo <pguo@pivotal.io> wrote:
> > I updated the patches as attached following previous discussions.
>
> Hi Paul,
>
> Could we please have a fresh rebase now that the CF is here?
>
> Thanks,
>
> --
> Thomas Munro
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__enterprisedb.com&d=DwIBaQ&c=lnl9vOaLMzs...
>
Attachments: download all (3 files, .tar.gz)
v3-0001-Extact-common-functions-from-pg_basebackup-into-s.patch (application/octet-stream, 16.9K) download
From 39bae0d58c7304e618cd06ec7a2722902473334a Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 18:55:09 +0800
Subject: [PATCH v3 1/3] Extact common functions from pg_basebackup into
separate files for pg_rewind use.
---
src/bin/pg_basebackup/Makefile | 2 +-
src/bin/pg_basebackup/backup_common.c | 176 ++++++++++++++++++++++++++
src/bin/pg_basebackup/backup_common.h | 33 +++++
src/bin/pg_basebackup/pg_basebackup.c | 174 +------------------------
src/bin/pg_basebackup/streamutil.c | 1 -
src/bin/pg_rewind/Makefile | 16 ++-
src/bin/pg_rewind/libpq_fetch.c | 2 +-
src/bin/pg_rewind/pg_rewind.c | 4 +
8 files changed, 230 insertions(+), 178 deletions(-)
create mode 100644 src/bin/pg_basebackup/backup_common.c
create mode 100644 src/bin/pg_basebackup/backup_common.h
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index d7a081b9bb..2ed77bd0d6 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -21,7 +21,7 @@ include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
-OBJS=receivelog.o streamutil.o walmethods.o $(WIN32RES)
+OBJS=receivelog.o streamutil.o walmethods.o backup_common.o $(WIN32RES)
all: pg_basebackup pg_receivewal pg_recvlogical
diff --git a/src/bin/pg_basebackup/backup_common.c b/src/bin/pg_basebackup/backup_common.c
new file mode 100644
index 0000000000..a5e541a152
--- /dev/null
+++ b/src/bin/pg_basebackup/backup_common.c
@@ -0,0 +1,176 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_common.c - common code that is used in both pg_basebackup and pg_rewind.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+#include "common/logging.h"
+#include "fe_utils/string_utils.h"
+#include "backup_common.h"
+
+/* Contents of configuration file to be generated */
+PQExpBuffer recoveryconfcontents = NULL;
+
+bool writerecoveryconf = false;
+char *replication_slot = NULL;
+PGconn *conn = NULL;
+
+void
+disconnect_atexit(void)
+{
+ if (conn != NULL)
+ PQfinish(conn);
+}
+
+/*
+ * Escape a string so that it can be used as a value in a key-value pair
+ * a configuration file.
+ */
+static char *
+escape_quotes(const char *src)
+{
+ char *result = escape_single_quotes_ascii(src);
+
+ if (!result)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+ return result;
+}
+
+/*
+ * Create a configuration file in memory using a PQExpBuffer
+ */
+void
+GenerateRecoveryConf(void)
+{
+ PQconninfoOption *connOptions;
+ PQconninfoOption *option;
+ PQExpBufferData conninfo_buf;
+ char *escaped;
+
+ recoveryconfcontents = createPQExpBuffer();
+ if (!recoveryconfcontents)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ /*
+ * In PostgreSQL 12 and newer versions, standby_mode is gone, replaced by
+ * standby.signal to trigger a standby state at recovery.
+ */
+ if (conn && PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
+ appendPQExpBufferStr(recoveryconfcontents, "standby_mode = 'on'\n");
+
+ connOptions = PQconninfo(conn);
+ if (connOptions == NULL)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ initPQExpBuffer(&conninfo_buf);
+ for (option = connOptions; option && option->keyword; option++)
+ {
+ /* Omit empty settings and those libpqwalreceiver overrides. */
+ if (strcmp(option->keyword, "replication") == 0 ||
+ strcmp(option->keyword, "dbname") == 0 ||
+ strcmp(option->keyword, "fallback_application_name") == 0 ||
+ (option->val == NULL) ||
+ (option->val != NULL && option->val[0] == '\0'))
+ continue;
+
+ /* Separate key-value pairs with spaces */
+ if (conninfo_buf.len != 0)
+ appendPQExpBufferChar(&conninfo_buf, ' ');
+
+ /*
+ * Write "keyword=value" pieces, the value string is escaped and/or
+ * quoted if necessary.
+ */
+ appendPQExpBuffer(&conninfo_buf, "%s=", option->keyword);
+ appendConnStrVal(&conninfo_buf, option->val);
+ }
+
+ /*
+ * Escape the connection string, so that it can be put in the config file.
+ * Note that this is different from the escaping of individual connection
+ * options above!
+ */
+ escaped = escape_quotes(conninfo_buf.data);
+ appendPQExpBuffer(recoveryconfcontents, "primary_conninfo = '%s'\n", escaped);
+ free(escaped);
+
+ if (replication_slot)
+ {
+ escaped = escape_quotes(replication_slot);
+ appendPQExpBuffer(recoveryconfcontents, "primary_slot_name = '%s'\n", replication_slot);
+ free(escaped);
+ }
+
+ if (PQExpBufferBroken(recoveryconfcontents) ||
+ PQExpBufferDataBroken(conninfo_buf))
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ termPQExpBuffer(&conninfo_buf);
+
+ PQconninfoFree(connOptions);
+}
+
+/*
+ * Write the configuration file into the directory specified in basedir,
+ * with the contents already collected in memory appended. Then write
+ * the signal file into the basedir. If the server does not support
+ * recovery parameters as GUCs, the signal file is not necessary, and
+ * configuration is written to recovery.conf.
+ */
+void
+WriteRecoveryConf(char *target_dir)
+{
+ char filename[MAXPGPATH];
+ FILE *cf;
+ bool is_recovery_guc_supported = true;
+
+ if (conn && PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
+ is_recovery_guc_supported = false;
+
+ snprintf(filename, MAXPGPATH, "%s/%s", target_dir,
+ is_recovery_guc_supported ? "postgresql.auto.conf" : "recovery.conf");
+
+ cf = fopen(filename, is_recovery_guc_supported ? "a" : "w");
+ if (cf == NULL)
+ {
+ pg_log_error("could not open file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ if (fwrite(recoveryconfcontents->data, recoveryconfcontents->len, 1, cf) != 1)
+ {
+ pg_log_error("could not write to file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ fclose(cf);
+
+ if (is_recovery_guc_supported)
+ {
+ snprintf(filename, MAXPGPATH, "%s/%s", target_dir, "standby.signal");
+ cf = fopen(filename, "w");
+ if (cf == NULL)
+ {
+ pg_log_error("could not create file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ fclose(cf);
+ }
+}
diff --git a/src/bin/pg_basebackup/backup_common.h b/src/bin/pg_basebackup/backup_common.h
new file mode 100644
index 0000000000..8f9f860485
--- /dev/null
+++ b/src/bin/pg_basebackup/backup_common.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_common.h - common code that is used in both pg_basebackup and pg_rewind.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef BACKUP_COMMON_H
+#define BACKUP_COMMON_H
+
+#include "libpq-fe.h"
+#include "pqexpbuffer.h"
+
+/* Contents of configuration file to be generated */
+extern PQExpBuffer recoveryconfcontents;
+
+extern bool writerecoveryconf;
+extern char *replication_slot;
+PGconn *conn;
+
+extern void disconnect_atexit(void);
+extern void GenerateRecoveryConf(void);
+extern void WriteRecoveryConf(char *target_dir);
+
+/*
+ * recovery.conf is integrated into postgresql.conf from version 12.
+ */
+#define MINIMUM_VERSION_FOR_RECOVERY_GUC 120000
+
+
+#endif /* BACKUP_COMMON_H */
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 63f554307c..28de306c75 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -37,6 +37,7 @@
#include "pqexpbuffer.h"
#include "pgtar.h"
#include "pgtime.h"
+#include "backup_common.h"
#include "receivelog.h"
#include "replication/basebackup.h"
#include "streamutil.h"
@@ -67,11 +68,6 @@ typedef struct TablespaceList
*/
#define MINIMUM_VERSION_FOR_TEMP_SLOTS 100000
-/*
- * recovery.conf is integrated into postgresql.conf from version 12.
- */
-#define MINIMUM_VERSION_FOR_RECOVERY_GUC 120000
-
/*
* Different ways to include WAL
*/
@@ -95,12 +91,10 @@ static int verbose = 0;
static int compresslevel = 0;
static IncludeWal includewal = STREAM_WAL;
static bool fastcheckpoint = false;
-static bool writerecoveryconf = false;
static bool do_sync = true;
static int standby_message_timeout = 10 * 1000; /* 10 sec = default */
static pg_time_t last_progress_report = 0;
static int32 maxrate = 0; /* no limit by default */
-static char *replication_slot = NULL;
static bool temp_replication_slot = true;
static bool create_slot = false;
static bool no_slot = false;
@@ -137,9 +131,6 @@ static int has_xlogendptr = 0;
static volatile LONG has_xlogendptr = 0;
#endif
-/* Contents of configuration file to be generated */
-static PQExpBuffer recoveryconfcontents = NULL;
-
/* Function headers */
static void usage(void);
static void verify_dir_is_empty_or_create(char *dirname, bool *created, bool *found);
@@ -147,8 +138,6 @@ static void progress_report(int tablespacenum, const char *filename, bool force)
static void ReceiveTarFile(PGconn *conn, PGresult *res, int rownum);
static void ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum);
-static void GenerateRecoveryConf(PGconn *conn);
-static void WriteRecoveryConf(void);
static void BaseBackup(void);
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
@@ -205,13 +194,6 @@ cleanup_directories_atexit(void)
pg_log_info("changes to tablespace directories will not be undone");
}
-static void
-disconnect_atexit(void)
-{
- if (conn != NULL)
- PQfinish(conn);
-}
-
#ifndef WIN32
/*
* On windows, our background thread dies along with the process. But on
@@ -1621,7 +1603,7 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
PQfreemem(copybuf);
if (basetablespace && writerecoveryconf)
- WriteRecoveryConf();
+ WriteRecoveryConf(basedir);
/*
* No data is synced here, everything is done for all tablespaces at the
@@ -1629,156 +1611,6 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
*/
}
-/*
- * Escape a string so that it can be used as a value in a key-value pair
- * a configuration file.
- */
-static char *
-escape_quotes(const char *src)
-{
- char *result = escape_single_quotes_ascii(src);
-
- if (!result)
- {
- pg_log_error("out of memory");
- exit(1);
- }
- return result;
-}
-
-/*
- * Create a configuration file in memory using a PQExpBuffer
- */
-static void
-GenerateRecoveryConf(PGconn *conn)
-{
- PQconninfoOption *connOptions;
- PQconninfoOption *option;
- PQExpBufferData conninfo_buf;
- char *escaped;
-
- recoveryconfcontents = createPQExpBuffer();
- if (!recoveryconfcontents)
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- /*
- * In PostgreSQL 12 and newer versions, standby_mode is gone, replaced by
- * standby.signal to trigger a standby state at recovery.
- */
- if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
- appendPQExpBufferStr(recoveryconfcontents, "standby_mode = 'on'\n");
-
- connOptions = PQconninfo(conn);
- if (connOptions == NULL)
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- initPQExpBuffer(&conninfo_buf);
- for (option = connOptions; option && option->keyword; option++)
- {
- /* Omit empty settings and those libpqwalreceiver overrides. */
- if (strcmp(option->keyword, "replication") == 0 ||
- strcmp(option->keyword, "dbname") == 0 ||
- strcmp(option->keyword, "fallback_application_name") == 0 ||
- (option->val == NULL) ||
- (option->val != NULL && option->val[0] == '\0'))
- continue;
-
- /* Separate key-value pairs with spaces */
- if (conninfo_buf.len != 0)
- appendPQExpBufferChar(&conninfo_buf, ' ');
-
- /*
- * Write "keyword=value" pieces, the value string is escaped and/or
- * quoted if necessary.
- */
- appendPQExpBuffer(&conninfo_buf, "%s=", option->keyword);
- appendConnStrVal(&conninfo_buf, option->val);
- }
-
- /*
- * Escape the connection string, so that it can be put in the config file.
- * Note that this is different from the escaping of individual connection
- * options above!
- */
- escaped = escape_quotes(conninfo_buf.data);
- appendPQExpBuffer(recoveryconfcontents, "primary_conninfo = '%s'\n", escaped);
- free(escaped);
-
- if (replication_slot)
- {
- escaped = escape_quotes(replication_slot);
- appendPQExpBuffer(recoveryconfcontents, "primary_slot_name = '%s'\n", replication_slot);
- free(escaped);
- }
-
- if (PQExpBufferBroken(recoveryconfcontents) ||
- PQExpBufferDataBroken(conninfo_buf))
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- termPQExpBuffer(&conninfo_buf);
-
- PQconninfoFree(connOptions);
-}
-
-
-/*
- * Write the configuration file into the directory specified in basedir,
- * with the contents already collected in memory appended. Then write
- * the signal file into the basedir. If the server does not support
- * recovery parameters as GUCs, the signal file is not necessary, and
- * configuration is written to recovery.conf.
- */
-static void
-WriteRecoveryConf(void)
-{
- char filename[MAXPGPATH];
- FILE *cf;
- bool is_recovery_guc_supported = true;
-
- if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
- is_recovery_guc_supported = false;
-
- snprintf(filename, MAXPGPATH, "%s/%s", basedir,
- is_recovery_guc_supported ? "postgresql.auto.conf" : "recovery.conf");
-
- cf = fopen(filename, is_recovery_guc_supported ? "a" : "w");
- if (cf == NULL)
- {
- pg_log_error("could not open file \"%s\": %m", filename);
- exit(1);
- }
-
- if (fwrite(recoveryconfcontents->data, recoveryconfcontents->len, 1, cf) != 1)
- {
- pg_log_error("could not write to file \"%s\": %m", filename);
- exit(1);
- }
-
- fclose(cf);
-
- if (is_recovery_guc_supported)
- {
- snprintf(filename, MAXPGPATH, "%s/%s", basedir, "standby.signal");
- cf = fopen(filename, "w");
- if (cf == NULL)
- {
- pg_log_error("could not create file \"%s\": %m", filename);
- exit(1);
- }
-
- fclose(cf);
- }
-}
-
static void
BaseBackup(void)
@@ -1835,7 +1667,7 @@ BaseBackup(void)
* Build contents of configuration file if requested
*/
if (writerecoveryconf)
- GenerateRecoveryConf(conn);
+ GenerateRecoveryConf();
/*
* Run IDENTIFY_SYSTEM so we can get the timeline
diff --git a/src/bin/pg_basebackup/streamutil.c b/src/bin/pg_basebackup/streamutil.c
index 79f17e4089..4bba197632 100644
--- a/src/bin/pg_basebackup/streamutil.c
+++ b/src/bin/pg_basebackup/streamutil.c
@@ -53,7 +53,6 @@ char *dbname = NULL;
int dbgetpassword = 0; /* 0=auto, -1=never, 1=always */
static bool have_password = false;
static char password[100];
-PGconn *conn = NULL;
/*
* Connect to the server. Returns a valid PGconn pointer if connected,
diff --git a/src/bin/pg_rewind/Makefile b/src/bin/pg_rewind/Makefile
index 859d3abc41..85f4182da0 100644
--- a/src/bin/pg_rewind/Makefile
+++ b/src/bin/pg_rewind/Makefile
@@ -16,19 +16,27 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I$(libpq_srcdir) -DFRONTEND $(CPPFLAGS)
-LDFLAGS_INTERNAL += $(libpq_pgport)
+LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
OBJS = pg_rewind.o parsexlog.o xlogreader.o datapagemap.o timeline.o \
- fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o \
+ fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o backup_common.o \
$(WIN32RES)
-EXTRA_CLEAN = xlogreader.c
+EXTRA_CLEAN = xlogreader.c backup_common.c backup_common.h
all: pg_rewind
pg_rewind: $(OBJS) | submake-libpq submake-libpgport
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_rewind.o: backup_common.h
+
+backup_common.c: % : $(top_srcdir)/src/bin/pg_basebackup/%
+ rm -f $@ && $(LN_S) $< .
+
+backup_common.h: % : $(top_srcdir)/src/bin/pg_basebackup/%
+ rm -f $@ && $(LN_S) $< .
+
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
@@ -42,7 +50,7 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_rewind$(X)'
clean distclean maintainer-clean:
- rm -f pg_rewind$(X) $(OBJS) xlogreader.c
+ rm -f pg_rewind$(X) $(OBJS) xlogreader.c backup_common.c backup_common.h
rm -rf tmp_check
check:
diff --git a/src/bin/pg_rewind/libpq_fetch.c b/src/bin/pg_rewind/libpq_fetch.c
index d6cbe23926..7a546ebe60 100644
--- a/src/bin/pg_rewind/libpq_fetch.c
+++ b/src/bin/pg_rewind/libpq_fetch.c
@@ -25,7 +25,7 @@
#include "fe_utils/connect.h"
#include "port/pg_bswap.h"
-static PGconn *conn = NULL;
+extern PGconn *conn;
/*
* Files are fetched max CHUNKSIZE bytes at a time.
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index 6e77201be6..14260bb883 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -29,6 +29,7 @@
#include "common/restricted_token.h"
#include "getopt_long.h"
#include "storage/bufpage.h"
+#include "backup_common.h"
static void usage(const char *progname);
@@ -231,7 +232,10 @@ main(int argc, char **argv)
/* Connect to remote server */
if (connstr_source)
+ {
libpqConnect(connstr_source);
+ atexit(disconnect_atexit);
+ }
/*
* Ok, we have all the options and we're ready to start. Read in all the
--
2.17.2
v3-0002-Add-option-to-write-recovery-configuration-inform.patch (application/octet-stream, 2.5K) download
From ebbc2975e07822e49dc5a53c1fa107d3354ad44c Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 19:37:36 +0800
Subject: [PATCH v3 2/3] Add option to write recovery configuration information
in pg_rewind.
---
src/bin/pg_rewind/pg_rewind.c | 19 ++++++++++++++++++-
1 file changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index 14260bb883..b31a193e5a 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -76,6 +76,7 @@ usage(const char *progname)
printf(_(" -D, --target-pgdata=DIRECTORY existing data directory to modify\n"));
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
+ printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -93,6 +94,7 @@ main(int argc, char **argv)
static struct option long_options[] = {
{"help", no_argument, NULL, '?'},
{"target-pgdata", required_argument, NULL, 'D'},
+ {"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
{"version", no_argument, NULL, 'V'},
@@ -135,7 +137,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNP", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -155,6 +157,10 @@ main(int argc, char **argv)
do_sync = false;
break;
+ case 'R':
+ writerecoveryconf = true;
+ break;
+
case 3:
debug = true;
pg_logging_set_level(PG_LOG_DEBUG);
@@ -301,6 +307,11 @@ main(int argc, char **argv)
if (!rewind_needed)
{
pg_log_info("no rewind required");
+ if (writerecoveryconf && connstr_source)
+ {
+ GenerateRecoveryConf();
+ WriteRecoveryConf(datadir_target);
+ }
exit(0);
}
@@ -398,6 +409,12 @@ main(int argc, char **argv)
pg_log_info("syncing target data directory");
syncTargetDirectory();
+ if (writerecoveryconf && connstr_source)
+ {
+ GenerateRecoveryConf();
+ WriteRecoveryConf(datadir_target);
+ }
+
pg_log_info("Done!");
return 0;
--
2.17.2
v3-0003-Ensure-target-clean-shutdown-at-the-beginning-of-.patch (application/octet-stream, 4.7K) download
From 350fc0e43142c5e37fcfc36af771c300be37b966 Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Tue, 19 Mar 2019 12:41:30 +0800
Subject: [PATCH v3 3/3] Ensure target clean shutdown at the beginning of
pg_rewind
---
src/bin/pg_rewind/pg_rewind.c | 70 ++++++++++++++++++++++++++++++++++-
1 file changed, 69 insertions(+), 1 deletion(-)
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index b31a193e5a..8f89f85e0a 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -41,6 +41,7 @@ static void digestControlFile(ControlFileData *ControlFile, char *source,
static void syncTargetDirectory(void);
static void sanityChecks(void);
static void findCommonAncestorTimeline(XLogRecPtr *recptr, int *tliIndex);
+static void ensureCleanShutdown(const char *argv0);
static ControlFileData ControlFile_target;
static ControlFileData ControlFile_source;
@@ -77,6 +78,7 @@ usage(const char *progname)
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
+ printf(_(" -e, --ensure-clean-shutdown run single-mode postgres if needed to make sure target is clean shutdown\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -97,6 +99,7 @@ main(int argc, char **argv)
{"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
+ {"ensure-clean-shutdown", no_argument, NULL, 'e'},
{"version", no_argument, NULL, 'V'},
{"dry-run", no_argument, NULL, 'n'},
{"no-sync", no_argument, NULL, 'N'},
@@ -113,6 +116,7 @@ main(int argc, char **argv)
XLogRecPtr chkptredo;
size_t size;
char *buffer;
+ bool ensure_clean_shutdown = false;
bool rewind_needed;
XLogRecPtr endrec;
TimeLineID endtli;
@@ -137,7 +141,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPRe", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -145,6 +149,10 @@ main(int argc, char **argv)
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
+ case 'e':
+ ensure_clean_shutdown = true;
+ break;
+
case 'P':
showprogress = true;
break;
@@ -251,6 +259,22 @@ main(int argc, char **argv)
digestControlFile(&ControlFile_target, buffer, size);
pg_free(buffer);
+ /*
+ * If the target instance was not cleanly shut down, run a single-user
+ * postgres session really quickly and reload the control file to get the
+ * new state.
+ */
+ if (ensure_clean_shutdown &&
+ ControlFile_target.state != DB_SHUTDOWNED &&
+ ControlFile_target.state != DB_SHUTDOWNED_IN_RECOVERY)
+ {
+ ensureCleanShutdown(argv[0]);
+
+ buffer = slurpFile(datadir_target, "global/pg_control", &size);
+ digestControlFile(&ControlFile_target, buffer, size);
+ pg_free(buffer);
+ }
+
buffer = fetchFile("global/pg_control", &size);
digestControlFile(&ControlFile_source, buffer, size);
pg_free(buffer);
@@ -769,3 +793,47 @@ syncTargetDirectory(void)
fsync_pgdata(datadir_target, PG_VERSION_NUM);
}
+
+/*
+ * Ensure clean shutdown of target instance by launching single-user mode
+ * postgres to do crash recovery.
+ */
+static void
+ensureCleanShutdown(const char *argv0)
+{
+ int ret;
+#define MAXCMDLEN (2 * MAXPGPATH)
+ char exec_path[MAXPGPATH];
+ char cmd[MAXCMDLEN];
+
+ /* locate postgres binary */
+ if ((ret = find_other_exec(argv0, "postgres",
+ PG_BACKEND_VERSIONSTR,
+ exec_path)) < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(argv0, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+
+ if (ret == -1)
+ pg_fatal("The program \"postgres\" is needed by %s but was \n"
+ "not found in the same directory as \"%s\".\n"
+ "Check your installation.\n", progname, full_path);
+ else
+ pg_fatal("The program \"postgres\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.\n", full_path, progname);
+ }
+
+ /* only skip processing after ensuring presence of postgres */
+ if (dry_run)
+ return;
+
+ /* finally run postgres single-user mode */
+ snprintf(cmd, MAXCMDLEN, "\"%s\" --single -D \"%s\" template1 < %s",
+ exec_path, datadir_target, DEVNULL);
+
+ if (system(cmd) != 0)
+ pg_fatal("postgres single-user mode of target instance failed for command: %s\n", cmd);
+}
--
2.17.2
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-07-08 22:54 ` Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
0 siblings, 1 reply; 46+ messages in thread
From: Thomas Munro @ 2019-07-08 22:54 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On Tue, Jul 2, 2019 at 5:46 PM Paul Guo <pguo@pivotal.io> wrote:
> Rebased, aligned with recent changes in pg_rewind/pg_basebackup and then retested. Thanks.
Hi Paul,
A minor build problem on Windows:
src/bin/pg_rewind/pg_rewind.c(32): fatal error C1083: Cannot open
include file: 'backup_common.h': No such file or directory
[C:\projects\postgresql\pg_rewind.vcxproj]
https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.46422
http://cfbot.cputube.org/paul-guo.html
--
Thomas Munro
https://enterprisedb.com
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
@ 2019-07-09 14:48 ` Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 46+ messages in thread
From: Paul Guo @ 2019-07-09 14:48 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
Yes, the patches changed Makefile so that pg_rewind and pg_basebackup could
use some common code, but for Windows build, I'm not sure where are those
window build files. Does anyone know about that? Thanks.
On Tue, Jul 9, 2019 at 6:55 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Tue, Jul 2, 2019 at 5:46 PM Paul Guo <pguo@pivotal.io> wrote:
> > Rebased, aligned with recent changes in pg_rewind/pg_basebackup and then
> retested. Thanks.
>
> Hi Paul,
>
> A minor build problem on Windows:
>
> src/bin/pg_rewind/pg_rewind.c(32): fatal error C1083: Cannot open
> include file: 'backup_common.h': No such file or directory
> [C:\projects\postgresql\pg_rewind.vcxproj]
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__ci.appveyor.com_project_postgresql-2Dcfbot_post...
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__cfbot.cputube.org_paul-2Dguo.html&d=DwIBaQ&a...
>
> --
> Thomas Munro
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__enterprisedb.com&d=DwIBaQ&c=lnl9vOaLMzs...
>
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-07-10 07:28 ` Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
0 siblings, 1 reply; 46+ messages in thread
From: Michael Paquier @ 2019-07-10 07:28 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On Tue, Jul 09, 2019 at 10:48:49PM +0800, Paul Guo wrote:
> Yes, the patches changed Makefile so that pg_rewind and pg_basebackup could
> use some common code, but for Windows build, I'm not sure where are those
> window build files. Does anyone know about that? Thanks.
The VS scripts are located in src/tools/msvc/. You will likely need
to tweak things like $frontend_extraincludes or variables in the same
area for this patch (please see Mkvcbuild.pm).
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
@ 2019-07-15 08:52 ` Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 46+ messages in thread
From: Paul Guo @ 2019-07-15 08:52 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On Wed, Jul 10, 2019 at 3:28 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Tue, Jul 09, 2019 at 10:48:49PM +0800, Paul Guo wrote:
> > Yes, the patches changed Makefile so that pg_rewind and pg_basebackup
> could
> > use some common code, but for Windows build, I'm not sure where are those
> > window build files. Does anyone know about that? Thanks.
>
> The VS scripts are located in src/tools/msvc/. You will likely need
> to tweak things like $frontend_extraincludes or variables in the same
> area for this patch (please see Mkvcbuild.pm).
>
Thanks. Both Mkvcbuild.pm and pg_rewind/Makefile are modified to make
Windows build pass in a
local environment (Hopefully this passes the CI testing), also now
pg_rewind/Makefile does not
create soft link for backup_common.h anymore. Instead -I is used to specify
the header directory.
I also noticed that doc change is needed so modified documents for the two
new options accordingly.
Please see the attached new patches.
Attachments: download all (3 files, .tar.gz)
v4-0002-Add-option-to-write-recovery-configuration-inform.patch (application/octet-stream, 3.3K) download
From d021b6ce3e2978167c9ad2bd4045275c32619c67 Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 19:37:36 +0800
Subject: [PATCH v4 2/3] Add option to write recovery configuration information
in pg_rewind.
---
doc/src/sgml/ref/pg_rewind.sgml | 11 +++++++++++
src/bin/pg_rewind/pg_rewind.c | 19 ++++++++++++++++++-
2 files changed, 29 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 52a1caa246..6a46b8c3b4 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -165,6 +165,17 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-R</option></term>
+ <term><option>--write-recovery-conf</option></term>
+ <listitem>
+ <para>
+ Create <filename>standby.signal</filename> and append connection settings
+ to <filename>postgresql.auto.conf</filename> in the output directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-n</option></term>
<term><option>--dry-run</option></term>
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index c3403d0a04..9d0f0b9282 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -76,6 +76,7 @@ usage(const char *progname)
printf(_(" -D, --target-pgdata=DIRECTORY existing data directory to modify\n"));
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
+ printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -93,6 +94,7 @@ main(int argc, char **argv)
static struct option long_options[] = {
{"help", no_argument, NULL, '?'},
{"target-pgdata", required_argument, NULL, 'D'},
+ {"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
{"version", no_argument, NULL, 'V'},
@@ -135,7 +137,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNP", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -155,6 +157,10 @@ main(int argc, char **argv)
do_sync = false;
break;
+ case 'R':
+ writerecoveryconf = true;
+ break;
+
case 3:
debug = true;
pg_logging_set_level(PG_LOG_DEBUG);
@@ -301,6 +307,11 @@ main(int argc, char **argv)
if (!rewind_needed)
{
pg_log_info("no rewind required");
+ if (writerecoveryconf && connstr_source)
+ {
+ GenerateRecoveryConf();
+ WriteRecoveryConf(datadir_target);
+ }
exit(0);
}
@@ -398,6 +409,12 @@ main(int argc, char **argv)
pg_log_info("syncing target data directory");
syncTargetDirectory();
+ if (writerecoveryconf && connstr_source)
+ {
+ GenerateRecoveryConf();
+ WriteRecoveryConf(datadir_target);
+ }
+
pg_log_info("Done!");
return 0;
--
2.17.2
v4-0003-Ensure-target-clean-shutdown-at-the-beginning-of-.patch (application/octet-stream, 5.9K) download
From 0bdd69a94973b4d3f2964626f4740394be3a6177 Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Tue, 19 Mar 2019 12:41:30 +0800
Subject: [PATCH v4 3/3] Ensure target clean shutdown at the beginning of
pg_rewind
---
doc/src/sgml/ref/pg_rewind.sgml | 17 ++++++++
src/bin/pg_rewind/pg_rewind.c | 70 ++++++++++++++++++++++++++++++++-
2 files changed, 86 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 6a46b8c3b4..8423a69f66 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -176,6 +176,23 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-e</option></term>
+ <term><option>--ensure-clean-shutdown</option></term>
+ <listitem>
+ <para>
+ Make sure target is clean shutdown before running
+ <application>pg_rewind</application>. With this option specified,
+ <application>pg_rewind</application> will launch a single mode
+ postgres instance if needed to ensure the target has completed crash
+ recovery and make sure pg_control is in clean shutdown state.
+ Clean shutdown state is a prerequisite of
+ <application>pg_rewind</application>. Without this option, users need
+ to do this manually.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-n</option></term>
<term><option>--dry-run</option></term>
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index 9d0f0b9282..bdce38f21f 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -41,6 +41,7 @@ static void digestControlFile(ControlFileData *ControlFile, char *source,
static void syncTargetDirectory(void);
static void sanityChecks(void);
static void findCommonAncestorTimeline(XLogRecPtr *recptr, int *tliIndex);
+static void ensureCleanShutdown(const char *argv0);
static ControlFileData ControlFile_target;
static ControlFileData ControlFile_source;
@@ -77,6 +78,7 @@ usage(const char *progname)
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
+ printf(_(" -e, --ensure-clean-shutdown run single-mode postgres if needed to make sure target is clean shutdown\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -97,6 +99,7 @@ main(int argc, char **argv)
{"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
+ {"ensure-clean-shutdown", no_argument, NULL, 'e'},
{"version", no_argument, NULL, 'V'},
{"dry-run", no_argument, NULL, 'n'},
{"no-sync", no_argument, NULL, 'N'},
@@ -113,6 +116,7 @@ main(int argc, char **argv)
XLogRecPtr chkptredo;
size_t size;
char *buffer;
+ bool ensure_clean_shutdown = false;
bool rewind_needed;
XLogRecPtr endrec;
TimeLineID endtli;
@@ -137,7 +141,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPRe", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -145,6 +149,10 @@ main(int argc, char **argv)
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
+ case 'e':
+ ensure_clean_shutdown = true;
+ break;
+
case 'P':
showprogress = true;
break;
@@ -251,6 +259,22 @@ main(int argc, char **argv)
digestControlFile(&ControlFile_target, buffer, size);
pg_free(buffer);
+ /*
+ * If the target instance was not cleanly shut down, run a single-user
+ * postgres session really quickly and reload the control file to get the
+ * new state.
+ */
+ if (ensure_clean_shutdown &&
+ ControlFile_target.state != DB_SHUTDOWNED &&
+ ControlFile_target.state != DB_SHUTDOWNED_IN_RECOVERY)
+ {
+ ensureCleanShutdown(argv[0]);
+
+ buffer = slurpFile(datadir_target, "global/pg_control", &size);
+ digestControlFile(&ControlFile_target, buffer, size);
+ pg_free(buffer);
+ }
+
buffer = fetchFile("global/pg_control", &size);
digestControlFile(&ControlFile_source, buffer, size);
pg_free(buffer);
@@ -769,3 +793,47 @@ syncTargetDirectory(void)
fsync_pgdata(datadir_target, PG_VERSION_NUM);
}
+
+/*
+ * Ensure clean shutdown of target instance by launching single-user mode
+ * postgres to do crash recovery.
+ */
+static void
+ensureCleanShutdown(const char *argv0)
+{
+ int ret;
+#define MAXCMDLEN (2 * MAXPGPATH)
+ char exec_path[MAXPGPATH];
+ char cmd[MAXCMDLEN];
+
+ /* locate postgres binary */
+ if ((ret = find_other_exec(argv0, "postgres",
+ PG_BACKEND_VERSIONSTR,
+ exec_path)) < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(argv0, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+
+ if (ret == -1)
+ pg_fatal("The program \"postgres\" is needed by %s but was \n"
+ "not found in the same directory as \"%s\".\n"
+ "Check your installation.\n", progname, full_path);
+ else
+ pg_fatal("The program \"postgres\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.\n", full_path, progname);
+ }
+
+ /* only skip processing after ensuring presence of postgres */
+ if (dry_run)
+ return;
+
+ /* finally run postgres single-user mode */
+ snprintf(cmd, MAXCMDLEN, "\"%s\" --single -D \"%s\" template1 < %s",
+ exec_path, datadir_target, DEVNULL);
+
+ if (system(cmd) != 0)
+ pg_fatal("postgres single-user mode of target instance failed for command: %s\n", cmd);
+}
--
2.17.2
v4-0001-Extact-common-functions-from-pg_basebackup-into-s.patch (application/octet-stream, 17.4K) download
From 3b852336cdac49c0f5fdb67421311463bbf6fc1d Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 18:55:09 +0800
Subject: [PATCH v4 1/3] Extact common functions from pg_basebackup into
separate files for pg_rewind use.
---
src/bin/pg_basebackup/Makefile | 2 +-
src/bin/pg_basebackup/backup_common.c | 176 ++++++++++++++++++++++++++
src/bin/pg_basebackup/backup_common.h | 33 +++++
src/bin/pg_basebackup/pg_basebackup.c | 174 +------------------------
src/bin/pg_basebackup/streamutil.c | 1 -
src/bin/pg_rewind/Makefile | 13 +-
src/bin/pg_rewind/libpq_fetch.c | 2 +-
src/bin/pg_rewind/pg_rewind.c | 4 +
src/tools/msvc/Mkvcbuild.pm | 2 +
9 files changed, 228 insertions(+), 179 deletions(-)
create mode 100644 src/bin/pg_basebackup/backup_common.c
create mode 100644 src/bin/pg_basebackup/backup_common.h
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index d7a081b9bb..2ed77bd0d6 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -21,7 +21,7 @@ include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
-OBJS=receivelog.o streamutil.o walmethods.o $(WIN32RES)
+OBJS=receivelog.o streamutil.o walmethods.o backup_common.o $(WIN32RES)
all: pg_basebackup pg_receivewal pg_recvlogical
diff --git a/src/bin/pg_basebackup/backup_common.c b/src/bin/pg_basebackup/backup_common.c
new file mode 100644
index 0000000000..a5e541a152
--- /dev/null
+++ b/src/bin/pg_basebackup/backup_common.c
@@ -0,0 +1,176 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_common.c - common code that is used in both pg_basebackup and pg_rewind.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+#include "common/logging.h"
+#include "fe_utils/string_utils.h"
+#include "backup_common.h"
+
+/* Contents of configuration file to be generated */
+PQExpBuffer recoveryconfcontents = NULL;
+
+bool writerecoveryconf = false;
+char *replication_slot = NULL;
+PGconn *conn = NULL;
+
+void
+disconnect_atexit(void)
+{
+ if (conn != NULL)
+ PQfinish(conn);
+}
+
+/*
+ * Escape a string so that it can be used as a value in a key-value pair
+ * a configuration file.
+ */
+static char *
+escape_quotes(const char *src)
+{
+ char *result = escape_single_quotes_ascii(src);
+
+ if (!result)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+ return result;
+}
+
+/*
+ * Create a configuration file in memory using a PQExpBuffer
+ */
+void
+GenerateRecoveryConf(void)
+{
+ PQconninfoOption *connOptions;
+ PQconninfoOption *option;
+ PQExpBufferData conninfo_buf;
+ char *escaped;
+
+ recoveryconfcontents = createPQExpBuffer();
+ if (!recoveryconfcontents)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ /*
+ * In PostgreSQL 12 and newer versions, standby_mode is gone, replaced by
+ * standby.signal to trigger a standby state at recovery.
+ */
+ if (conn && PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
+ appendPQExpBufferStr(recoveryconfcontents, "standby_mode = 'on'\n");
+
+ connOptions = PQconninfo(conn);
+ if (connOptions == NULL)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ initPQExpBuffer(&conninfo_buf);
+ for (option = connOptions; option && option->keyword; option++)
+ {
+ /* Omit empty settings and those libpqwalreceiver overrides. */
+ if (strcmp(option->keyword, "replication") == 0 ||
+ strcmp(option->keyword, "dbname") == 0 ||
+ strcmp(option->keyword, "fallback_application_name") == 0 ||
+ (option->val == NULL) ||
+ (option->val != NULL && option->val[0] == '\0'))
+ continue;
+
+ /* Separate key-value pairs with spaces */
+ if (conninfo_buf.len != 0)
+ appendPQExpBufferChar(&conninfo_buf, ' ');
+
+ /*
+ * Write "keyword=value" pieces, the value string is escaped and/or
+ * quoted if necessary.
+ */
+ appendPQExpBuffer(&conninfo_buf, "%s=", option->keyword);
+ appendConnStrVal(&conninfo_buf, option->val);
+ }
+
+ /*
+ * Escape the connection string, so that it can be put in the config file.
+ * Note that this is different from the escaping of individual connection
+ * options above!
+ */
+ escaped = escape_quotes(conninfo_buf.data);
+ appendPQExpBuffer(recoveryconfcontents, "primary_conninfo = '%s'\n", escaped);
+ free(escaped);
+
+ if (replication_slot)
+ {
+ escaped = escape_quotes(replication_slot);
+ appendPQExpBuffer(recoveryconfcontents, "primary_slot_name = '%s'\n", replication_slot);
+ free(escaped);
+ }
+
+ if (PQExpBufferBroken(recoveryconfcontents) ||
+ PQExpBufferDataBroken(conninfo_buf))
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ termPQExpBuffer(&conninfo_buf);
+
+ PQconninfoFree(connOptions);
+}
+
+/*
+ * Write the configuration file into the directory specified in basedir,
+ * with the contents already collected in memory appended. Then write
+ * the signal file into the basedir. If the server does not support
+ * recovery parameters as GUCs, the signal file is not necessary, and
+ * configuration is written to recovery.conf.
+ */
+void
+WriteRecoveryConf(char *target_dir)
+{
+ char filename[MAXPGPATH];
+ FILE *cf;
+ bool is_recovery_guc_supported = true;
+
+ if (conn && PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
+ is_recovery_guc_supported = false;
+
+ snprintf(filename, MAXPGPATH, "%s/%s", target_dir,
+ is_recovery_guc_supported ? "postgresql.auto.conf" : "recovery.conf");
+
+ cf = fopen(filename, is_recovery_guc_supported ? "a" : "w");
+ if (cf == NULL)
+ {
+ pg_log_error("could not open file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ if (fwrite(recoveryconfcontents->data, recoveryconfcontents->len, 1, cf) != 1)
+ {
+ pg_log_error("could not write to file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ fclose(cf);
+
+ if (is_recovery_guc_supported)
+ {
+ snprintf(filename, MAXPGPATH, "%s/%s", target_dir, "standby.signal");
+ cf = fopen(filename, "w");
+ if (cf == NULL)
+ {
+ pg_log_error("could not create file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ fclose(cf);
+ }
+}
diff --git a/src/bin/pg_basebackup/backup_common.h b/src/bin/pg_basebackup/backup_common.h
new file mode 100644
index 0000000000..8f9f860485
--- /dev/null
+++ b/src/bin/pg_basebackup/backup_common.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_common.h - common code that is used in both pg_basebackup and pg_rewind.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef BACKUP_COMMON_H
+#define BACKUP_COMMON_H
+
+#include "libpq-fe.h"
+#include "pqexpbuffer.h"
+
+/* Contents of configuration file to be generated */
+extern PQExpBuffer recoveryconfcontents;
+
+extern bool writerecoveryconf;
+extern char *replication_slot;
+PGconn *conn;
+
+extern void disconnect_atexit(void);
+extern void GenerateRecoveryConf(void);
+extern void WriteRecoveryConf(char *target_dir);
+
+/*
+ * recovery.conf is integrated into postgresql.conf from version 12.
+ */
+#define MINIMUM_VERSION_FOR_RECOVERY_GUC 120000
+
+
+#endif /* BACKUP_COMMON_H */
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 15f43f9432..cdcdc4daf8 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -37,6 +37,7 @@
#include "pqexpbuffer.h"
#include "pgtar.h"
#include "pgtime.h"
+#include "backup_common.h"
#include "receivelog.h"
#include "replication/basebackup.h"
#include "streamutil.h"
@@ -67,11 +68,6 @@ typedef struct TablespaceList
*/
#define MINIMUM_VERSION_FOR_TEMP_SLOTS 100000
-/*
- * recovery.conf is integrated into postgresql.conf from version 12.
- */
-#define MINIMUM_VERSION_FOR_RECOVERY_GUC 120000
-
/*
* Different ways to include WAL
*/
@@ -95,12 +91,10 @@ static int verbose = 0;
static int compresslevel = 0;
static IncludeWal includewal = STREAM_WAL;
static bool fastcheckpoint = false;
-static bool writerecoveryconf = false;
static bool do_sync = true;
static int standby_message_timeout = 10 * 1000; /* 10 sec = default */
static pg_time_t last_progress_report = 0;
static int32 maxrate = 0; /* no limit by default */
-static char *replication_slot = NULL;
static bool temp_replication_slot = true;
static bool create_slot = false;
static bool no_slot = false;
@@ -137,9 +131,6 @@ static int has_xlogendptr = 0;
static volatile LONG has_xlogendptr = 0;
#endif
-/* Contents of configuration file to be generated */
-static PQExpBuffer recoveryconfcontents = NULL;
-
/* Function headers */
static void usage(void);
static void verify_dir_is_empty_or_create(char *dirname, bool *created, bool *found);
@@ -147,8 +138,6 @@ static void progress_report(int tablespacenum, const char *filename, bool force)
static void ReceiveTarFile(PGconn *conn, PGresult *res, int rownum);
static void ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum);
-static void GenerateRecoveryConf(PGconn *conn);
-static void WriteRecoveryConf(void);
static void BaseBackup(void);
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
@@ -205,13 +194,6 @@ cleanup_directories_atexit(void)
pg_log_info("changes to tablespace directories will not be undone");
}
-static void
-disconnect_atexit(void)
-{
- if (conn != NULL)
- PQfinish(conn);
-}
-
#ifndef WIN32
/*
* On windows, our background thread dies along with the process. But on
@@ -1621,7 +1603,7 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
PQfreemem(copybuf);
if (basetablespace && writerecoveryconf)
- WriteRecoveryConf();
+ WriteRecoveryConf(basedir);
/*
* No data is synced here, everything is done for all tablespaces at the
@@ -1629,156 +1611,6 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
*/
}
-/*
- * Escape a string so that it can be used as a value in a key-value pair
- * a configuration file.
- */
-static char *
-escape_quotes(const char *src)
-{
- char *result = escape_single_quotes_ascii(src);
-
- if (!result)
- {
- pg_log_error("out of memory");
- exit(1);
- }
- return result;
-}
-
-/*
- * Create a configuration file in memory using a PQExpBuffer
- */
-static void
-GenerateRecoveryConf(PGconn *conn)
-{
- PQconninfoOption *connOptions;
- PQconninfoOption *option;
- PQExpBufferData conninfo_buf;
- char *escaped;
-
- recoveryconfcontents = createPQExpBuffer();
- if (!recoveryconfcontents)
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- /*
- * In PostgreSQL 12 and newer versions, standby_mode is gone, replaced by
- * standby.signal to trigger a standby state at recovery.
- */
- if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
- appendPQExpBufferStr(recoveryconfcontents, "standby_mode = 'on'\n");
-
- connOptions = PQconninfo(conn);
- if (connOptions == NULL)
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- initPQExpBuffer(&conninfo_buf);
- for (option = connOptions; option && option->keyword; option++)
- {
- /* Omit empty settings and those libpqwalreceiver overrides. */
- if (strcmp(option->keyword, "replication") == 0 ||
- strcmp(option->keyword, "dbname") == 0 ||
- strcmp(option->keyword, "fallback_application_name") == 0 ||
- (option->val == NULL) ||
- (option->val != NULL && option->val[0] == '\0'))
- continue;
-
- /* Separate key-value pairs with spaces */
- if (conninfo_buf.len != 0)
- appendPQExpBufferChar(&conninfo_buf, ' ');
-
- /*
- * Write "keyword=value" pieces, the value string is escaped and/or
- * quoted if necessary.
- */
- appendPQExpBuffer(&conninfo_buf, "%s=", option->keyword);
- appendConnStrVal(&conninfo_buf, option->val);
- }
-
- /*
- * Escape the connection string, so that it can be put in the config file.
- * Note that this is different from the escaping of individual connection
- * options above!
- */
- escaped = escape_quotes(conninfo_buf.data);
- appendPQExpBuffer(recoveryconfcontents, "primary_conninfo = '%s'\n", escaped);
- free(escaped);
-
- if (replication_slot)
- {
- escaped = escape_quotes(replication_slot);
- appendPQExpBuffer(recoveryconfcontents, "primary_slot_name = '%s'\n", replication_slot);
- free(escaped);
- }
-
- if (PQExpBufferBroken(recoveryconfcontents) ||
- PQExpBufferDataBroken(conninfo_buf))
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- termPQExpBuffer(&conninfo_buf);
-
- PQconninfoFree(connOptions);
-}
-
-
-/*
- * Write the configuration file into the directory specified in basedir,
- * with the contents already collected in memory appended. Then write
- * the signal file into the basedir. If the server does not support
- * recovery parameters as GUCs, the signal file is not necessary, and
- * configuration is written to recovery.conf.
- */
-static void
-WriteRecoveryConf(void)
-{
- char filename[MAXPGPATH];
- FILE *cf;
- bool is_recovery_guc_supported = true;
-
- if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
- is_recovery_guc_supported = false;
-
- snprintf(filename, MAXPGPATH, "%s/%s", basedir,
- is_recovery_guc_supported ? "postgresql.auto.conf" : "recovery.conf");
-
- cf = fopen(filename, is_recovery_guc_supported ? "a" : "w");
- if (cf == NULL)
- {
- pg_log_error("could not open file \"%s\": %m", filename);
- exit(1);
- }
-
- if (fwrite(recoveryconfcontents->data, recoveryconfcontents->len, 1, cf) != 1)
- {
- pg_log_error("could not write to file \"%s\": %m", filename);
- exit(1);
- }
-
- fclose(cf);
-
- if (is_recovery_guc_supported)
- {
- snprintf(filename, MAXPGPATH, "%s/%s", basedir, "standby.signal");
- cf = fopen(filename, "w");
- if (cf == NULL)
- {
- pg_log_error("could not create file \"%s\": %m", filename);
- exit(1);
- }
-
- fclose(cf);
- }
-}
-
static void
BaseBackup(void)
@@ -1835,7 +1667,7 @@ BaseBackup(void)
* Build contents of configuration file if requested
*/
if (writerecoveryconf)
- GenerateRecoveryConf(conn);
+ GenerateRecoveryConf();
/*
* Run IDENTIFY_SYSTEM so we can get the timeline
diff --git a/src/bin/pg_basebackup/streamutil.c b/src/bin/pg_basebackup/streamutil.c
index 8d8ac11b66..7c78835268 100644
--- a/src/bin/pg_basebackup/streamutil.c
+++ b/src/bin/pg_basebackup/streamutil.c
@@ -53,7 +53,6 @@ char *dbname = NULL;
int dbgetpassword = 0; /* 0=auto, -1=never, 1=always */
static bool have_password = false;
static char password[100];
-PGconn *conn = NULL;
/*
* Connect to the server. Returns a valid PGconn pointer if connected,
diff --git a/src/bin/pg_rewind/Makefile b/src/bin/pg_rewind/Makefile
index 859d3abc41..555f5cb31f 100644
--- a/src/bin/pg_rewind/Makefile
+++ b/src/bin/pg_rewind/Makefile
@@ -15,20 +15,23 @@ subdir = src/bin/pg_rewind
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-override CPPFLAGS := -I$(libpq_srcdir) -DFRONTEND $(CPPFLAGS)
-LDFLAGS_INTERNAL += $(libpq_pgport)
+override CPPFLAGS := -I$(libpq_srcdir) -DFRONTEND -I$(top_srcdir)/src/bin/pg_basebackup $(CPPFLAGS)
+LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
OBJS = pg_rewind.o parsexlog.o xlogreader.o datapagemap.o timeline.o \
- fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o \
+ fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o backup_common.o \
$(WIN32RES)
-EXTRA_CLEAN = xlogreader.c
+EXTRA_CLEAN = xlogreader.c backup_common.c
all: pg_rewind
pg_rewind: $(OBJS) | submake-libpq submake-libpgport
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+backup_common.c: % : $(top_srcdir)/src/bin/pg_basebackup/%
+ rm -f $@ && $(LN_S) $< .
+
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
@@ -42,7 +45,7 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_rewind$(X)'
clean distclean maintainer-clean:
- rm -f pg_rewind$(X) $(OBJS) xlogreader.c
+ rm -f pg_rewind$(X) $(OBJS) xlogreader.c backup_common.c
rm -rf tmp_check
check:
diff --git a/src/bin/pg_rewind/libpq_fetch.c b/src/bin/pg_rewind/libpq_fetch.c
index 37eccc3126..6e13aa5c04 100644
--- a/src/bin/pg_rewind/libpq_fetch.c
+++ b/src/bin/pg_rewind/libpq_fetch.c
@@ -25,7 +25,7 @@
#include "fe_utils/connect.h"
#include "port/pg_bswap.h"
-static PGconn *conn = NULL;
+extern PGconn *conn;
/*
* Files are fetched max CHUNKSIZE bytes at a time.
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index d378053de4..c3403d0a04 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -29,6 +29,7 @@
#include "common/restricted_token.h"
#include "getopt_long.h"
#include "storage/bufpage.h"
+#include "backup_common.h"
static void usage(const char *progname);
@@ -231,7 +232,10 @@ main(int argc, char **argv)
/* Connect to remote server */
if (connstr_source)
+ {
libpqConnect(connstr_source);
+ atexit(disconnect_atexit);
+ }
/*
* Ok, we have all the options and we're ready to start. Read in all the
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index d1d0aed07e..cf6beb938a 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -375,7 +375,9 @@ sub mkvcbuild
my $pgrewind = AddSimpleFrontend('pg_rewind', 1);
$pgrewind->{name} = 'pg_rewind';
+ $pgrewind->AddIncludeDir('src/bin/pg_basebackup');
$pgrewind->AddFile('src/backend/access/transam/xlogreader.c');
+ $pgrewind->AddFile('src/bin/pg_basebackup/backup_common.c');
$pgrewind->AddLibrary('ws2_32.lib');
$pgrewind->AddDefine('FRONTEND');
--
2.17.2
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-09-03 22:39 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
0 siblings, 1 reply; 46+ messages in thread
From: Alvaro Herrera @ 2019-09-03 22:39 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On 2019-Jul-15, Paul Guo wrote:
> Thanks. Both Mkvcbuild.pm and pg_rewind/Makefile are modified to make
> Windows build pass in a
> local environment (Hopefully this passes the CI testing), also now
> pg_rewind/Makefile does not
> create soft link for backup_common.h anymore. Instead -I is used to specify
> the header directory.
It seems there's minor breakage in the build, per CFbot. Can you
please rebase this?
Thanks
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2019-09-05 07:41 ` Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 46+ messages in thread
From: Paul Guo @ 2019-09-05 07:41 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
>
> It seems there's minor breakage in the build, per CFbot. Can you
> please rebase this?
>
> There is a code conflict. See attached for the new version. Thanks.
Attachments: download all (3 files, .tar.gz)
v5-0001-Extact-common-functions-from-pg_basebackup-into-s.patch (application/octet-stream, 17.5K) download
From fabcc0a53c88e4ac595780088c05af686070a445 Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 18:55:09 +0800
Subject: [PATCH v5 1/3] Extact common functions from pg_basebackup into
separate files for pg_rewind use.
---
src/bin/pg_basebackup/Makefile | 2 +-
src/bin/pg_basebackup/backup_common.c | 176 ++++++++++++++++++++++++++
src/bin/pg_basebackup/backup_common.h | 33 +++++
src/bin/pg_basebackup/pg_basebackup.c | 174 +------------------------
src/bin/pg_basebackup/streamutil.c | 1 -
src/bin/pg_rewind/Makefile | 13 +-
src/bin/pg_rewind/libpq_fetch.c | 2 +-
src/bin/pg_rewind/pg_rewind.c | 4 +
src/tools/msvc/Mkvcbuild.pm | 2 +
9 files changed, 228 insertions(+), 179 deletions(-)
create mode 100644 src/bin/pg_basebackup/backup_common.c
create mode 100644 src/bin/pg_basebackup/backup_common.h
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index d7a081b9bb..2ed77bd0d6 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -21,7 +21,7 @@ include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
-OBJS=receivelog.o streamutil.o walmethods.o $(WIN32RES)
+OBJS=receivelog.o streamutil.o walmethods.o backup_common.o $(WIN32RES)
all: pg_basebackup pg_receivewal pg_recvlogical
diff --git a/src/bin/pg_basebackup/backup_common.c b/src/bin/pg_basebackup/backup_common.c
new file mode 100644
index 0000000000..89a377630b
--- /dev/null
+++ b/src/bin/pg_basebackup/backup_common.c
@@ -0,0 +1,176 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_common.c - common code that is used in both pg_basebackup and pg_rewind.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+#include "common/logging.h"
+#include "fe_utils/string_utils.h"
+#include "backup_common.h"
+
+/* Contents of configuration file to be generated */
+PQExpBuffer recoveryconfcontents = NULL;
+
+bool writerecoveryconf = false;
+char *replication_slot = NULL;
+PGconn *conn = NULL;
+
+void
+disconnect_atexit(void)
+{
+ if (conn != NULL)
+ PQfinish(conn);
+}
+
+/*
+ * Escape a string so that it can be used as a value in a key-value pair
+ * a configuration file.
+ */
+static char *
+escape_quotes(const char *src)
+{
+ char *result = escape_single_quotes_ascii(src);
+
+ if (!result)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+ return result;
+}
+
+/*
+ * Create a configuration file in memory using a PQExpBuffer
+ */
+void
+GenerateRecoveryConf(void)
+{
+ PQconninfoOption *connOptions;
+ PQconninfoOption *option;
+ PQExpBufferData conninfo_buf;
+ char *escaped;
+
+ recoveryconfcontents = createPQExpBuffer();
+ if (!recoveryconfcontents)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ /*
+ * In PostgreSQL 12 and newer versions, standby_mode is gone, replaced by
+ * standby.signal to trigger a standby state at recovery.
+ */
+ if (conn && PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
+ appendPQExpBufferStr(recoveryconfcontents, "standby_mode = 'on'\n");
+
+ connOptions = PQconninfo(conn);
+ if (connOptions == NULL)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ initPQExpBuffer(&conninfo_buf);
+ for (option = connOptions; option && option->keyword; option++)
+ {
+ /* Omit empty settings and those libpqwalreceiver overrides. */
+ if (strcmp(option->keyword, "replication") == 0 ||
+ strcmp(option->keyword, "dbname") == 0 ||
+ strcmp(option->keyword, "fallback_application_name") == 0 ||
+ (option->val == NULL) ||
+ (option->val != NULL && option->val[0] == '\0'))
+ continue;
+
+ /* Separate key-value pairs with spaces */
+ if (conninfo_buf.len != 0)
+ appendPQExpBufferChar(&conninfo_buf, ' ');
+
+ /*
+ * Write "keyword=value" pieces, the value string is escaped and/or
+ * quoted if necessary.
+ */
+ appendPQExpBuffer(&conninfo_buf, "%s=", option->keyword);
+ appendConnStrVal(&conninfo_buf, option->val);
+ }
+
+ /*
+ * Escape the connection string, so that it can be put in the config file.
+ * Note that this is different from the escaping of individual connection
+ * options above!
+ */
+ escaped = escape_quotes(conninfo_buf.data);
+ appendPQExpBuffer(recoveryconfcontents, "primary_conninfo = '%s'\n", escaped);
+ free(escaped);
+
+ if (replication_slot)
+ {
+ /* unescaped: ReplicationSlotValidateName allows [a-z0-9_] only */
+ appendPQExpBuffer(recoveryconfcontents, "primary_slot_name = '%s'\n",
+ replication_slot);
+ }
+
+ if (PQExpBufferBroken(recoveryconfcontents) ||
+ PQExpBufferDataBroken(conninfo_buf))
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ termPQExpBuffer(&conninfo_buf);
+
+ PQconninfoFree(connOptions);
+}
+
+/*
+ * Write the configuration file into the directory specified in basedir,
+ * with the contents already collected in memory appended. Then write
+ * the signal file into the basedir. If the server does not support
+ * recovery parameters as GUCs, the signal file is not necessary, and
+ * configuration is written to recovery.conf.
+ */
+void
+WriteRecoveryConf(char *target_dir)
+{
+ char filename[MAXPGPATH];
+ FILE *cf;
+ bool is_recovery_guc_supported = true;
+
+ if (conn && PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
+ is_recovery_guc_supported = false;
+
+ snprintf(filename, MAXPGPATH, "%s/%s", target_dir,
+ is_recovery_guc_supported ? "postgresql.auto.conf" : "recovery.conf");
+
+ cf = fopen(filename, is_recovery_guc_supported ? "a" : "w");
+ if (cf == NULL)
+ {
+ pg_log_error("could not open file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ if (fwrite(recoveryconfcontents->data, recoveryconfcontents->len, 1, cf) != 1)
+ {
+ pg_log_error("could not write to file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ fclose(cf);
+
+ if (is_recovery_guc_supported)
+ {
+ snprintf(filename, MAXPGPATH, "%s/%s", target_dir, "standby.signal");
+ cf = fopen(filename, "w");
+ if (cf == NULL)
+ {
+ pg_log_error("could not create file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ fclose(cf);
+ }
+}
diff --git a/src/bin/pg_basebackup/backup_common.h b/src/bin/pg_basebackup/backup_common.h
new file mode 100644
index 0000000000..8f9f860485
--- /dev/null
+++ b/src/bin/pg_basebackup/backup_common.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * backup_common.h - common code that is used in both pg_basebackup and pg_rewind.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef BACKUP_COMMON_H
+#define BACKUP_COMMON_H
+
+#include "libpq-fe.h"
+#include "pqexpbuffer.h"
+
+/* Contents of configuration file to be generated */
+extern PQExpBuffer recoveryconfcontents;
+
+extern bool writerecoveryconf;
+extern char *replication_slot;
+PGconn *conn;
+
+extern void disconnect_atexit(void);
+extern void GenerateRecoveryConf(void);
+extern void WriteRecoveryConf(char *target_dir);
+
+/*
+ * recovery.conf is integrated into postgresql.conf from version 12.
+ */
+#define MINIMUM_VERSION_FOR_RECOVERY_GUC 120000
+
+
+#endif /* BACKUP_COMMON_H */
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 7986872f10..169fc2429c 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -37,6 +37,7 @@
#include "pqexpbuffer.h"
#include "pgtar.h"
#include "pgtime.h"
+#include "backup_common.h"
#include "receivelog.h"
#include "replication/basebackup.h"
#include "streamutil.h"
@@ -67,11 +68,6 @@ typedef struct TablespaceList
*/
#define MINIMUM_VERSION_FOR_TEMP_SLOTS 100000
-/*
- * recovery.conf is integrated into postgresql.conf from version 12.
- */
-#define MINIMUM_VERSION_FOR_RECOVERY_GUC 120000
-
/*
* Different ways to include WAL
*/
@@ -95,12 +91,10 @@ static int verbose = 0;
static int compresslevel = 0;
static IncludeWal includewal = STREAM_WAL;
static bool fastcheckpoint = false;
-static bool writerecoveryconf = false;
static bool do_sync = true;
static int standby_message_timeout = 10 * 1000; /* 10 sec = default */
static pg_time_t last_progress_report = 0;
static int32 maxrate = 0; /* no limit by default */
-static char *replication_slot = NULL;
static bool temp_replication_slot = true;
static bool create_slot = false;
static bool no_slot = false;
@@ -137,9 +131,6 @@ static int has_xlogendptr = 0;
static volatile LONG has_xlogendptr = 0;
#endif
-/* Contents of configuration file to be generated */
-static PQExpBuffer recoveryconfcontents = NULL;
-
/* Function headers */
static void usage(void);
static void verify_dir_is_empty_or_create(char *dirname, bool *created, bool *found);
@@ -147,8 +138,6 @@ static void progress_report(int tablespacenum, const char *filename, bool force)
static void ReceiveTarFile(PGconn *conn, PGresult *res, int rownum);
static void ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum);
-static void GenerateRecoveryConf(PGconn *conn);
-static void WriteRecoveryConf(void);
static void BaseBackup(void);
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
@@ -205,13 +194,6 @@ cleanup_directories_atexit(void)
pg_log_info("changes to tablespace directories will not be undone");
}
-static void
-disconnect_atexit(void)
-{
- if (conn != NULL)
- PQfinish(conn);
-}
-
#ifndef WIN32
/*
* On windows, our background thread dies along with the process. But on
@@ -1629,7 +1611,7 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
PQfreemem(copybuf);
if (basetablespace && writerecoveryconf)
- WriteRecoveryConf();
+ WriteRecoveryConf(basedir);
/*
* No data is synced here, everything is done for all tablespaces at the
@@ -1637,156 +1619,6 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
*/
}
-/*
- * Escape a string so that it can be used as a value in a key-value pair
- * a configuration file.
- */
-static char *
-escape_quotes(const char *src)
-{
- char *result = escape_single_quotes_ascii(src);
-
- if (!result)
- {
- pg_log_error("out of memory");
- exit(1);
- }
- return result;
-}
-
-/*
- * Create a configuration file in memory using a PQExpBuffer
- */
-static void
-GenerateRecoveryConf(PGconn *conn)
-{
- PQconninfoOption *connOptions;
- PQconninfoOption *option;
- PQExpBufferData conninfo_buf;
- char *escaped;
-
- recoveryconfcontents = createPQExpBuffer();
- if (!recoveryconfcontents)
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- /*
- * In PostgreSQL 12 and newer versions, standby_mode is gone, replaced by
- * standby.signal to trigger a standby state at recovery.
- */
- if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
- appendPQExpBufferStr(recoveryconfcontents, "standby_mode = 'on'\n");
-
- connOptions = PQconninfo(conn);
- if (connOptions == NULL)
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- initPQExpBuffer(&conninfo_buf);
- for (option = connOptions; option && option->keyword; option++)
- {
- /* Omit empty settings and those libpqwalreceiver overrides. */
- if (strcmp(option->keyword, "replication") == 0 ||
- strcmp(option->keyword, "dbname") == 0 ||
- strcmp(option->keyword, "fallback_application_name") == 0 ||
- (option->val == NULL) ||
- (option->val != NULL && option->val[0] == '\0'))
- continue;
-
- /* Separate key-value pairs with spaces */
- if (conninfo_buf.len != 0)
- appendPQExpBufferChar(&conninfo_buf, ' ');
-
- /*
- * Write "keyword=value" pieces, the value string is escaped and/or
- * quoted if necessary.
- */
- appendPQExpBuffer(&conninfo_buf, "%s=", option->keyword);
- appendConnStrVal(&conninfo_buf, option->val);
- }
-
- /*
- * Escape the connection string, so that it can be put in the config file.
- * Note that this is different from the escaping of individual connection
- * options above!
- */
- escaped = escape_quotes(conninfo_buf.data);
- appendPQExpBuffer(recoveryconfcontents, "primary_conninfo = '%s'\n", escaped);
- free(escaped);
-
- if (replication_slot)
- {
- /* unescaped: ReplicationSlotValidateName allows [a-z0-9_] only */
- appendPQExpBuffer(recoveryconfcontents, "primary_slot_name = '%s'\n",
- replication_slot);
- }
-
- if (PQExpBufferBroken(recoveryconfcontents) ||
- PQExpBufferDataBroken(conninfo_buf))
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- termPQExpBuffer(&conninfo_buf);
-
- PQconninfoFree(connOptions);
-}
-
-
-/*
- * Write the configuration file into the directory specified in basedir,
- * with the contents already collected in memory appended. Then write
- * the signal file into the basedir. If the server does not support
- * recovery parameters as GUCs, the signal file is not necessary, and
- * configuration is written to recovery.conf.
- */
-static void
-WriteRecoveryConf(void)
-{
- char filename[MAXPGPATH];
- FILE *cf;
- bool is_recovery_guc_supported = true;
-
- if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
- is_recovery_guc_supported = false;
-
- snprintf(filename, MAXPGPATH, "%s/%s", basedir,
- is_recovery_guc_supported ? "postgresql.auto.conf" : "recovery.conf");
-
- cf = fopen(filename, is_recovery_guc_supported ? "a" : "w");
- if (cf == NULL)
- {
- pg_log_error("could not open file \"%s\": %m", filename);
- exit(1);
- }
-
- if (fwrite(recoveryconfcontents->data, recoveryconfcontents->len, 1, cf) != 1)
- {
- pg_log_error("could not write to file \"%s\": %m", filename);
- exit(1);
- }
-
- fclose(cf);
-
- if (is_recovery_guc_supported)
- {
- snprintf(filename, MAXPGPATH, "%s/%s", basedir, "standby.signal");
- cf = fopen(filename, "w");
- if (cf == NULL)
- {
- pg_log_error("could not create file \"%s\": %m", filename);
- exit(1);
- }
-
- fclose(cf);
- }
-}
-
static void
BaseBackup(void)
@@ -1843,7 +1675,7 @@ BaseBackup(void)
* Build contents of configuration file if requested
*/
if (writerecoveryconf)
- GenerateRecoveryConf(conn);
+ GenerateRecoveryConf();
/*
* Run IDENTIFY_SYSTEM so we can get the timeline
diff --git a/src/bin/pg_basebackup/streamutil.c b/src/bin/pg_basebackup/streamutil.c
index 8d8ac11b66..7c78835268 100644
--- a/src/bin/pg_basebackup/streamutil.c
+++ b/src/bin/pg_basebackup/streamutil.c
@@ -53,7 +53,6 @@ char *dbname = NULL;
int dbgetpassword = 0; /* 0=auto, -1=never, 1=always */
static bool have_password = false;
static char password[100];
-PGconn *conn = NULL;
/*
* Connect to the server. Returns a valid PGconn pointer if connected,
diff --git a/src/bin/pg_rewind/Makefile b/src/bin/pg_rewind/Makefile
index 859d3abc41..555f5cb31f 100644
--- a/src/bin/pg_rewind/Makefile
+++ b/src/bin/pg_rewind/Makefile
@@ -15,20 +15,23 @@ subdir = src/bin/pg_rewind
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-override CPPFLAGS := -I$(libpq_srcdir) -DFRONTEND $(CPPFLAGS)
-LDFLAGS_INTERNAL += $(libpq_pgport)
+override CPPFLAGS := -I$(libpq_srcdir) -DFRONTEND -I$(top_srcdir)/src/bin/pg_basebackup $(CPPFLAGS)
+LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
OBJS = pg_rewind.o parsexlog.o xlogreader.o datapagemap.o timeline.o \
- fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o \
+ fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o backup_common.o \
$(WIN32RES)
-EXTRA_CLEAN = xlogreader.c
+EXTRA_CLEAN = xlogreader.c backup_common.c
all: pg_rewind
pg_rewind: $(OBJS) | submake-libpq submake-libpgport
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+backup_common.c: % : $(top_srcdir)/src/bin/pg_basebackup/%
+ rm -f $@ && $(LN_S) $< .
+
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
@@ -42,7 +45,7 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_rewind$(X)'
clean distclean maintainer-clean:
- rm -f pg_rewind$(X) $(OBJS) xlogreader.c
+ rm -f pg_rewind$(X) $(OBJS) xlogreader.c backup_common.c
rm -rf tmp_check
check:
diff --git a/src/bin/pg_rewind/libpq_fetch.c b/src/bin/pg_rewind/libpq_fetch.c
index 002776f696..2f41d56b3d 100644
--- a/src/bin/pg_rewind/libpq_fetch.c
+++ b/src/bin/pg_rewind/libpq_fetch.c
@@ -25,7 +25,7 @@
#include "fe_utils/connect.h"
#include "port/pg_bswap.h"
-static PGconn *conn = NULL;
+extern PGconn *conn;
/*
* Files are fetched max CHUNKSIZE bytes at a time.
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index 15e3eab550..e8674c24bb 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -29,6 +29,7 @@
#include "common/restricted_token.h"
#include "getopt_long.h"
#include "storage/bufpage.h"
+#include "backup_common.h"
static void usage(const char *progname);
@@ -231,7 +232,10 @@ main(int argc, char **argv)
/* Connect to remote server */
if (connstr_source)
+ {
libpqConnect(connstr_source);
+ atexit(disconnect_atexit);
+ }
/*
* Ok, we have all the options and we're ready to start. Read in all the
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 239f13cc12..0d4ae86c0f 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -375,7 +375,9 @@ sub mkvcbuild
my $pgrewind = AddSimpleFrontend('pg_rewind', 1);
$pgrewind->{name} = 'pg_rewind';
+ $pgrewind->AddIncludeDir('src/bin/pg_basebackup');
$pgrewind->AddFile('src/backend/access/transam/xlogreader.c');
+ $pgrewind->AddFile('src/bin/pg_basebackup/backup_common.c');
$pgrewind->AddLibrary('ws2_32.lib');
$pgrewind->AddDefine('FRONTEND');
--
2.17.2
v5-0002-Add-option-to-write-recovery-configuration-inform.patch (application/octet-stream, 3.3K) download
From f2357bfc5f1cb94ddb64323104bdaf8473480fdd Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 19:37:36 +0800
Subject: [PATCH v5 2/3] Add option to write recovery configuration information
in pg_rewind.
---
doc/src/sgml/ref/pg_rewind.sgml | 11 +++++++++++
src/bin/pg_rewind/pg_rewind.c | 19 ++++++++++++++++++-
2 files changed, 29 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 52a1caa246..6a46b8c3b4 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -165,6 +165,17 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-R</option></term>
+ <term><option>--write-recovery-conf</option></term>
+ <listitem>
+ <para>
+ Create <filename>standby.signal</filename> and append connection settings
+ to <filename>postgresql.auto.conf</filename> in the output directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-n</option></term>
<term><option>--dry-run</option></term>
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index e8674c24bb..48487d3448 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -76,6 +76,7 @@ usage(const char *progname)
printf(_(" -D, --target-pgdata=DIRECTORY existing data directory to modify\n"));
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
+ printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -93,6 +94,7 @@ main(int argc, char **argv)
static struct option long_options[] = {
{"help", no_argument, NULL, '?'},
{"target-pgdata", required_argument, NULL, 'D'},
+ {"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
{"version", no_argument, NULL, 'V'},
@@ -135,7 +137,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNP", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -155,6 +157,10 @@ main(int argc, char **argv)
do_sync = false;
break;
+ case 'R':
+ writerecoveryconf = true;
+ break;
+
case 3:
debug = true;
pg_logging_set_level(PG_LOG_DEBUG);
@@ -301,6 +307,11 @@ main(int argc, char **argv)
if (!rewind_needed)
{
pg_log_info("no rewind required");
+ if (writerecoveryconf && connstr_source)
+ {
+ GenerateRecoveryConf();
+ WriteRecoveryConf(datadir_target);
+ }
exit(0);
}
@@ -398,6 +409,12 @@ main(int argc, char **argv)
pg_log_info("syncing target data directory");
syncTargetDirectory();
+ if (writerecoveryconf && connstr_source)
+ {
+ GenerateRecoveryConf();
+ WriteRecoveryConf(datadir_target);
+ }
+
pg_log_info("Done!");
return 0;
--
2.17.2
v5-0003-Ensure-target-clean-shutdown-at-the-beginning-of-.patch (application/octet-stream, 5.9K) download
From 07420ef602925951d63360aa00331f86c6c259d8 Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Tue, 19 Mar 2019 12:41:30 +0800
Subject: [PATCH v5 3/3] Ensure target clean shutdown at the beginning of
pg_rewind
---
doc/src/sgml/ref/pg_rewind.sgml | 17 ++++++++
src/bin/pg_rewind/pg_rewind.c | 70 ++++++++++++++++++++++++++++++++-
2 files changed, 86 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 6a46b8c3b4..8423a69f66 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -176,6 +176,23 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-e</option></term>
+ <term><option>--ensure-clean-shutdown</option></term>
+ <listitem>
+ <para>
+ Make sure target is clean shutdown before running
+ <application>pg_rewind</application>. With this option specified,
+ <application>pg_rewind</application> will launch a single mode
+ postgres instance if needed to ensure the target has completed crash
+ recovery and make sure pg_control is in clean shutdown state.
+ Clean shutdown state is a prerequisite of
+ <application>pg_rewind</application>. Without this option, users need
+ to do this manually.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-n</option></term>
<term><option>--dry-run</option></term>
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index 48487d3448..5296c69af5 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -41,6 +41,7 @@ static void digestControlFile(ControlFileData *ControlFile, char *source,
static void syncTargetDirectory(void);
static void sanityChecks(void);
static void findCommonAncestorTimeline(XLogRecPtr *recptr, int *tliIndex);
+static void ensureCleanShutdown(const char *argv0);
static ControlFileData ControlFile_target;
static ControlFileData ControlFile_source;
@@ -77,6 +78,7 @@ usage(const char *progname)
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
+ printf(_(" -e, --ensure-clean-shutdown run single-mode postgres if needed to make sure target is clean shutdown\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -97,6 +99,7 @@ main(int argc, char **argv)
{"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
+ {"ensure-clean-shutdown", no_argument, NULL, 'e'},
{"version", no_argument, NULL, 'V'},
{"dry-run", no_argument, NULL, 'n'},
{"no-sync", no_argument, NULL, 'N'},
@@ -113,6 +116,7 @@ main(int argc, char **argv)
XLogRecPtr chkptredo;
size_t size;
char *buffer;
+ bool ensure_clean_shutdown = false;
bool rewind_needed;
XLogRecPtr endrec;
TimeLineID endtli;
@@ -137,7 +141,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPRe", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -145,6 +149,10 @@ main(int argc, char **argv)
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
+ case 'e':
+ ensure_clean_shutdown = true;
+ break;
+
case 'P':
showprogress = true;
break;
@@ -251,6 +259,22 @@ main(int argc, char **argv)
digestControlFile(&ControlFile_target, buffer, size);
pg_free(buffer);
+ /*
+ * If the target instance was not cleanly shut down, run a single-user
+ * postgres session really quickly and reload the control file to get the
+ * new state.
+ */
+ if (ensure_clean_shutdown &&
+ ControlFile_target.state != DB_SHUTDOWNED &&
+ ControlFile_target.state != DB_SHUTDOWNED_IN_RECOVERY)
+ {
+ ensureCleanShutdown(argv[0]);
+
+ buffer = slurpFile(datadir_target, "global/pg_control", &size);
+ digestControlFile(&ControlFile_target, buffer, size);
+ pg_free(buffer);
+ }
+
buffer = fetchFile("global/pg_control", &size);
digestControlFile(&ControlFile_source, buffer, size);
pg_free(buffer);
@@ -769,3 +793,47 @@ syncTargetDirectory(void)
fsync_pgdata(datadir_target, PG_VERSION_NUM);
}
+
+/*
+ * Ensure clean shutdown of target instance by launching single-user mode
+ * postgres to do crash recovery.
+ */
+static void
+ensureCleanShutdown(const char *argv0)
+{
+ int ret;
+#define MAXCMDLEN (2 * MAXPGPATH)
+ char exec_path[MAXPGPATH];
+ char cmd[MAXCMDLEN];
+
+ /* locate postgres binary */
+ if ((ret = find_other_exec(argv0, "postgres",
+ PG_BACKEND_VERSIONSTR,
+ exec_path)) < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(argv0, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+
+ if (ret == -1)
+ pg_fatal("The program \"postgres\" is needed by %s but was \n"
+ "not found in the same directory as \"%s\".\n"
+ "Check your installation.\n", progname, full_path);
+ else
+ pg_fatal("The program \"postgres\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.\n", full_path, progname);
+ }
+
+ /* only skip processing after ensuring presence of postgres */
+ if (dry_run)
+ return;
+
+ /* finally run postgres single-user mode */
+ snprintf(cmd, MAXCMDLEN, "\"%s\" --single -D \"%s\" template1 < %s",
+ exec_path, datadir_target, DEVNULL);
+
+ if (system(cmd) != 0)
+ pg_fatal("postgres single-user mode of target instance failed for command: %s\n", cmd);
+}
--
2.17.2
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-09-05 13:23 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
0 siblings, 1 reply; 46+ messages in thread
From: Alvaro Herrera @ 2019-09-05 13:23 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
Thank for rebasing.
I didn't like 0001 very much.
* It seems now would be the time to stop pretending we're using a file
called recovery.conf; I know we still support older server versions that
use that file, but it sounds like we should take the opportunity to
rename the function to be less misleading once those versions vanish out
of existance.
* disconnect_and_exit seems a bit out of place compared to the other
parts of this new module. I think you only put it there so that the
'conn' can be a global, and that you can stop passing 'conn' as a
variable to GenerateRecoveryConf. It seems more modular to me to keep
it as a separate variable in each program and have it passed down to the
routine that writes the file.
* From modularity also seems better to me to avoid a global variable
'recoveryconfcontents' and instead return the string from
GenerateRecoveryConf to pass as a param to WriteRecoveryConf.
(In fact, I wonder why the current structure is like it is, namely to
have ReceiveAndUnpackTarFile write the file; why wouldn't be its caller
be responsible for writing it?)
I wonder about putting this new file in src/fe_utils instead of keeping
it in pg_basebackup and symlinking to pg_rewind. Maybe if we make it a
true module (recovery_config_gen.c) it makes more sense there.
0002 seems okay as far as it goes.
0003:
I still don't understand why we need a command-line option to do this.
Why should it not be the default behavior?
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2019-09-09 14:18 ` Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 46+ messages in thread
From: Paul Guo @ 2019-09-09 14:18 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
>
> Thank for rebasing.
>
> I didn't like 0001 very much.
>
> * It seems now would be the time to stop pretending we're using a file
> called recovery.conf; I know we still support older server versions that
> use that file, but it sounds like we should take the opportunity to
> rename the function to be less misleading once those versions vanish out
> of existance.
>
How about renaming the function names to
GenerateRecoveryConf -> GenerateRecoveryConfContents
WriteRecoveryConf -> WriteRecoveryConfInfo <- it writes standby.signal
on pg12+, so function name WriteRecoveryConfContents is not accurate.
and
variable writerecoveryconf -> write_recovery_conf_info?
> * disconnect_and_exit seems a bit out of place compared to the other
> parts of this new module. I think you only put it there so that the
> 'conn' can be a global, and that you can stop passing 'conn' as a
> variable to GenerateRecoveryConf. It seems more modular to me to keep
> it as a separate variable in each program and have it passed down to the
> routine that writes the file.
>
> * From modularity also seems better to me to avoid a global variable
> 'recoveryconfcontents' and instead return the string from
> GenerateRecoveryConf to pass as a param to WriteRecoveryConf.
> (In fact, I wonder why the current structure is like it is, namely to
> have ReceiveAndUnpackTarFile write the file; why wouldn't be its caller
> be responsible for writing it?)
>
Reasonable to make common code include less variables. I can try modifying
the patches to remove the previously added variables below in the common
code.
+/* Contents of configuration file to be generated */
+extern PQExpBuffer recoveryconfcontents;
+
+extern bool writerecoveryconf;
+extern char *replication_slot;
+PGconn *conn;
>
> I wonder about putting this new file in src/fe_utils instead of keeping
> it in pg_basebackup and symlinking to pg_rewind. Maybe if we make it a
> true module (recovery_config_gen.c) it makes more sense there.
>
> I thought some about where to put the common code also. It seems pg_rewind
and pg_basebackup are the only consumers of the small common code. I doubt
it deserves a separate file under src/fe_utils.
>
> 0003:
>
> I still don't understand why we need a command-line option to do this.
> Why should it not be the default behavior?
>
This was discussed but frankly speaking I do not know how other postgres
users or enterprise providers handle this (probably some have own scripts?).
I could easily remove the option code if more and more people agree on that
or at least we could turn it on by default?
Thanks
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-09-10 17:52 ` Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
0 siblings, 1 reply; 46+ messages in thread
From: Alvaro Herrera from 2ndQuadrant @ 2019-09-10 17:52 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On 2019-Sep-09, Paul Guo wrote:
> >
> > Thank for rebasing.
> >
> > I didn't like 0001 very much.
> >
> > * It seems now would be the time to stop pretending we're using a file
> > called recovery.conf; I know we still support older server versions that
> > use that file, but it sounds like we should take the opportunity to
> > rename the function to be less misleading once those versions vanish out
> > of existance.
>
> How about renaming the function names to
> GenerateRecoveryConf -> GenerateRecoveryConfContents
> WriteRecoveryConf -> WriteRecoveryConfInfo <- it writes standby.signal
> on pg12+, so function name WriteRecoveryConfContents is not accurate.
GenerateRecoveryConfig / WriteRecoveryConfig ?
> > I wonder about putting this new file in src/fe_utils instead of keeping
> > it in pg_basebackup and symlinking to pg_rewind. Maybe if we make it a
> > true module (recovery_config_gen.c) it makes more sense there.
> >
> I thought some about where to put the common code also. It seems pg_rewind
> and pg_basebackup are the only consumers of the small common code. I doubt
> it deserves a separate file under src/fe_utils.
Hmm, but other things there are also used by only two programs, say
psqlscan.l and conditional.c are just for psql and pgbench.
> > 0003:
> >
> > I still don't understand why we need a command-line option to do this.
> > Why should it not be the default behavior?
>
> This was discussed but frankly speaking I do not know how other postgres
> users or enterprise providers handle this (probably some have own scripts?).
> I could easily remove the option code if more and more people agree on that
> or at least we could turn it on by default?
Well, I've seen no contrary votes, and frankly I see no use for the
opposite (current) behavior.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
@ 2019-09-19 13:21 ` Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
0 siblings, 1 reply; 46+ messages in thread
From: Paul Guo @ 2019-09-19 13:21 UTC (permalink / raw)
To: Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
I've updated the patch series following the suggestions. Thanks.
>
>
Attachments: download all (3 files, .tar.gz)
v6-0001-Extact-common-functions-from-pg_basebackup-into-s.patch (application/octet-stream, 14.2K) download
From 9f8883341075d44ab699b468e3cc8bf200deb72d Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 18:55:09 +0800
Subject: [PATCH v6 1/3] Extact common functions from pg_basebackup into
separate files for pg_rewind use.
---
src/bin/pg_basebackup/pg_basebackup.c | 162 +------------------------
src/bin/pg_rewind/libpq_fetch.c | 3 +-
src/bin/pg_rewind/pg_rewind.c | 9 ++
src/bin/pg_rewind/pg_rewind.h | 3 +
src/fe_utils/Makefile | 2 +-
src/fe_utils/recovery_gen.c | 165 ++++++++++++++++++++++++++
src/include/fe_utils/recovery_gen.h | 25 ++++
7 files changed, 207 insertions(+), 162 deletions(-)
create mode 100644 src/fe_utils/recovery_gen.c
create mode 100644 src/include/fe_utils/recovery_gen.h
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 7986872f10..55ef13926d 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -31,6 +31,7 @@
#include "common/file_utils.h"
#include "common/logging.h"
#include "common/string.h"
+#include "fe_utils/recovery_gen.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
#include "libpq-fe.h"
@@ -67,11 +68,6 @@ typedef struct TablespaceList
*/
#define MINIMUM_VERSION_FOR_TEMP_SLOTS 100000
-/*
- * recovery.conf is integrated into postgresql.conf from version 12.
- */
-#define MINIMUM_VERSION_FOR_RECOVERY_GUC 120000
-
/*
* Different ways to include WAL
*/
@@ -147,8 +143,6 @@ static void progress_report(int tablespacenum, const char *filename, bool force)
static void ReceiveTarFile(PGconn *conn, PGresult *res, int rownum);
static void ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum);
-static void GenerateRecoveryConf(PGconn *conn);
-static void WriteRecoveryConf(void);
static void BaseBackup(void);
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
@@ -1629,7 +1623,7 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
PQfreemem(copybuf);
if (basetablespace && writerecoveryconf)
- WriteRecoveryConf();
+ WriteRecoveryConfig(conn, basedir, recoveryconfcontents);
/*
* No data is synced here, everything is done for all tablespaces at the
@@ -1637,156 +1631,6 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
*/
}
-/*
- * Escape a string so that it can be used as a value in a key-value pair
- * a configuration file.
- */
-static char *
-escape_quotes(const char *src)
-{
- char *result = escape_single_quotes_ascii(src);
-
- if (!result)
- {
- pg_log_error("out of memory");
- exit(1);
- }
- return result;
-}
-
-/*
- * Create a configuration file in memory using a PQExpBuffer
- */
-static void
-GenerateRecoveryConf(PGconn *conn)
-{
- PQconninfoOption *connOptions;
- PQconninfoOption *option;
- PQExpBufferData conninfo_buf;
- char *escaped;
-
- recoveryconfcontents = createPQExpBuffer();
- if (!recoveryconfcontents)
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- /*
- * In PostgreSQL 12 and newer versions, standby_mode is gone, replaced by
- * standby.signal to trigger a standby state at recovery.
- */
- if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
- appendPQExpBufferStr(recoveryconfcontents, "standby_mode = 'on'\n");
-
- connOptions = PQconninfo(conn);
- if (connOptions == NULL)
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- initPQExpBuffer(&conninfo_buf);
- for (option = connOptions; option && option->keyword; option++)
- {
- /* Omit empty settings and those libpqwalreceiver overrides. */
- if (strcmp(option->keyword, "replication") == 0 ||
- strcmp(option->keyword, "dbname") == 0 ||
- strcmp(option->keyword, "fallback_application_name") == 0 ||
- (option->val == NULL) ||
- (option->val != NULL && option->val[0] == '\0'))
- continue;
-
- /* Separate key-value pairs with spaces */
- if (conninfo_buf.len != 0)
- appendPQExpBufferChar(&conninfo_buf, ' ');
-
- /*
- * Write "keyword=value" pieces, the value string is escaped and/or
- * quoted if necessary.
- */
- appendPQExpBuffer(&conninfo_buf, "%s=", option->keyword);
- appendConnStrVal(&conninfo_buf, option->val);
- }
-
- /*
- * Escape the connection string, so that it can be put in the config file.
- * Note that this is different from the escaping of individual connection
- * options above!
- */
- escaped = escape_quotes(conninfo_buf.data);
- appendPQExpBuffer(recoveryconfcontents, "primary_conninfo = '%s'\n", escaped);
- free(escaped);
-
- if (replication_slot)
- {
- /* unescaped: ReplicationSlotValidateName allows [a-z0-9_] only */
- appendPQExpBuffer(recoveryconfcontents, "primary_slot_name = '%s'\n",
- replication_slot);
- }
-
- if (PQExpBufferBroken(recoveryconfcontents) ||
- PQExpBufferDataBroken(conninfo_buf))
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- termPQExpBuffer(&conninfo_buf);
-
- PQconninfoFree(connOptions);
-}
-
-
-/*
- * Write the configuration file into the directory specified in basedir,
- * with the contents already collected in memory appended. Then write
- * the signal file into the basedir. If the server does not support
- * recovery parameters as GUCs, the signal file is not necessary, and
- * configuration is written to recovery.conf.
- */
-static void
-WriteRecoveryConf(void)
-{
- char filename[MAXPGPATH];
- FILE *cf;
- bool is_recovery_guc_supported = true;
-
- if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
- is_recovery_guc_supported = false;
-
- snprintf(filename, MAXPGPATH, "%s/%s", basedir,
- is_recovery_guc_supported ? "postgresql.auto.conf" : "recovery.conf");
-
- cf = fopen(filename, is_recovery_guc_supported ? "a" : "w");
- if (cf == NULL)
- {
- pg_log_error("could not open file \"%s\": %m", filename);
- exit(1);
- }
-
- if (fwrite(recoveryconfcontents->data, recoveryconfcontents->len, 1, cf) != 1)
- {
- pg_log_error("could not write to file \"%s\": %m", filename);
- exit(1);
- }
-
- fclose(cf);
-
- if (is_recovery_guc_supported)
- {
- snprintf(filename, MAXPGPATH, "%s/%s", basedir, "standby.signal");
- cf = fopen(filename, "w");
- if (cf == NULL)
- {
- pg_log_error("could not create file \"%s\": %m", filename);
- exit(1);
- }
-
- fclose(cf);
- }
-}
-
static void
BaseBackup(void)
@@ -1843,7 +1687,7 @@ BaseBackup(void)
* Build contents of configuration file if requested
*/
if (writerecoveryconf)
- GenerateRecoveryConf(conn);
+ recoveryconfcontents = GenerateRecoveryConfig(conn, replication_slot);
/*
* Run IDENTIFY_SYSTEM so we can get the timeline
diff --git a/src/bin/pg_rewind/libpq_fetch.c b/src/bin/pg_rewind/libpq_fetch.c
index 002776f696..f4ebf7d842 100644
--- a/src/bin/pg_rewind/libpq_fetch.c
+++ b/src/bin/pg_rewind/libpq_fetch.c
@@ -20,12 +20,11 @@
#include "file_ops.h"
#include "filemap.h"
-#include "libpq-fe.h"
#include "catalog/pg_type_d.h"
#include "fe_utils/connect.h"
#include "port/pg_bswap.h"
-static PGconn *conn = NULL;
+PGconn *conn = NULL;
/*
* Files are fetched max CHUNKSIZE bytes at a time.
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index 15e3eab550..c3903c5753 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -85,6 +85,12 @@ usage(const char *progname)
printf(_("\nReport bugs to <pgsql-bugs@lists.postgresql.org>.\n"));
}
+static void
+disconnect_atexit(void)
+{
+ if (conn != NULL)
+ PQfinish(conn);
+}
int
main(int argc, char **argv)
@@ -231,7 +237,10 @@ main(int argc, char **argv)
/* Connect to remote server */
if (connstr_source)
+ {
libpqConnect(connstr_source);
+ atexit(disconnect_atexit);
+ }
/*
* Ok, we have all the options and we're ready to start. Read in all the
diff --git a/src/bin/pg_rewind/pg_rewind.h b/src/bin/pg_rewind/pg_rewind.h
index 1125c7e60f..d23c855107 100644
--- a/src/bin/pg_rewind/pg_rewind.h
+++ b/src/bin/pg_rewind/pg_rewind.h
@@ -19,6 +19,8 @@
#include "common/logging.h"
+#include "libpq-fe.h"
+
/* Configuration options */
extern char *datadir_target;
extern char *datadir_source;
@@ -26,6 +28,7 @@ extern char *connstr_source;
extern bool showprogress;
extern bool dry_run;
extern int WalSegSz;
+extern PGconn *conn;
/* Target history */
extern TimeLineHistoryEntry *targetHistory;
diff --git a/src/fe_utils/Makefile b/src/fe_utils/Makefile
index 7d73800323..ae0f6c884b 100644
--- a/src/fe_utils/Makefile
+++ b/src/fe_utils/Makefile
@@ -19,7 +19,7 @@ include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -DFRONTEND -I$(libpq_srcdir) $(CPPFLAGS)
-OBJS = mbprint.o print.o psqlscan.o simple_list.o string_utils.o conditional.o
+OBJS = mbprint.o print.o psqlscan.o simple_list.o string_utils.o conditional.o recovery_gen.o
all: libpgfeutils.a
diff --git a/src/fe_utils/recovery_gen.c b/src/fe_utils/recovery_gen.c
new file mode 100644
index 0000000000..4e0f6a154a
--- /dev/null
+++ b/src/fe_utils/recovery_gen.c
@@ -0,0 +1,165 @@
+/*-------------------------------------------------------------------------
+ *
+ * recovery_gen.c - general code that is used in both pg_basebackup and pg_rewind.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+#include "common/logging.h"
+#include "fe_utils/string_utils.h"
+#include "fe_utils/recovery_gen.h"
+
+/*
+ * Escape a string so that it can be used as a value in a key-value pair
+ * a configuration file.
+ */
+static char *
+escape_quotes(const char *src)
+{
+ char *result = escape_single_quotes_ascii(src);
+
+ if (!result)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+ return result;
+}
+
+/*
+ * Create recovery configuration contents using a PQExpBuffer
+ */
+PQExpBuffer
+GenerateRecoveryConfig(PGconn *pgconn, char *pg_replication_slot)
+{
+ PQconninfoOption *connOptions;
+ PQconninfoOption *option;
+ PQExpBufferData conninfo_buf;
+ char *escaped;
+ PQExpBuffer contents;
+
+ contents = createPQExpBuffer();
+ if (!contents)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ /*
+ * In PostgreSQL 12 and newer versions, standby_mode is gone, replaced by
+ * standby.signal to trigger a standby state at recovery.
+ */
+ if (pgconn && PQserverVersion(pgconn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
+ appendPQExpBufferStr(contents, "standby_mode = 'on'\n");
+
+ connOptions = PQconninfo(pgconn);
+ if (connOptions == NULL)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ initPQExpBuffer(&conninfo_buf);
+ for (option = connOptions; option && option->keyword; option++)
+ {
+ /* Omit empty settings and those libpqwalreceiver overrides. */
+ if (strcmp(option->keyword, "replication") == 0 ||
+ strcmp(option->keyword, "dbname") == 0 ||
+ strcmp(option->keyword, "fallback_application_name") == 0 ||
+ (option->val == NULL) ||
+ (option->val != NULL && option->val[0] == '\0'))
+ continue;
+
+ /* Separate key-value pairs with spaces */
+ if (conninfo_buf.len != 0)
+ appendPQExpBufferChar(&conninfo_buf, ' ');
+
+ /*
+ * Write "keyword=value" pieces, the value string is escaped and/or
+ * quoted if necessary.
+ */
+ appendPQExpBuffer(&conninfo_buf, "%s=", option->keyword);
+ appendConnStrVal(&conninfo_buf, option->val);
+ }
+
+ /*
+ * Escape the connection string, so that it can be put in the config file.
+ * Note that this is different from the escaping of individual connection
+ * options above!
+ */
+ escaped = escape_quotes(conninfo_buf.data);
+ appendPQExpBuffer(contents, "primary_conninfo = '%s'\n", escaped);
+ free(escaped);
+
+ if (pg_replication_slot)
+ {
+ /* unescaped: ReplicationSlotValidateName allows [a-z0-9_] only */
+ appendPQExpBuffer(contents, "primary_slot_name = '%s'\n",
+ pg_replication_slot);
+ }
+
+ if (PQExpBufferBroken(contents) ||
+ PQExpBufferDataBroken(conninfo_buf))
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ termPQExpBuffer(&conninfo_buf);
+
+ PQconninfoFree(connOptions);
+
+ return contents;
+}
+
+/*
+ * Write the configuration file in the directory specified in target_dir,
+ * with the contents already collected in memory appended. Then write
+ * the signal file into the target_dir. If the server does not support
+ * recovery parameters as GUCs, the signal file is not necessary, and
+ * configuration is written to recovery.conf.
+ */
+void
+WriteRecoveryConfig(PGconn *pgconn, char *target_dir, PQExpBuffer contents)
+{
+ char filename[MAXPGPATH];
+ FILE *cf;
+ bool is_recovery_guc_supported = true;
+
+ if (pgconn && PQserverVersion(pgconn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
+ is_recovery_guc_supported = false;
+
+ snprintf(filename, MAXPGPATH, "%s/%s", target_dir,
+ is_recovery_guc_supported ? "postgresql.auto.conf" : "recovery.conf");
+
+ cf = fopen(filename, is_recovery_guc_supported ? "a" : "w");
+ if (cf == NULL)
+ {
+ pg_log_error("could not open file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ if (fwrite(contents->data, contents->len, 1, cf) != 1)
+ {
+ pg_log_error("could not write to file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ fclose(cf);
+
+ if (is_recovery_guc_supported)
+ {
+ snprintf(filename, MAXPGPATH, "%s/%s", target_dir, "standby.signal");
+ cf = fopen(filename, "w");
+ if (cf == NULL)
+ {
+ pg_log_error("could not create file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ fclose(cf);
+ }
+}
diff --git a/src/include/fe_utils/recovery_gen.h b/src/include/fe_utils/recovery_gen.h
new file mode 100644
index 0000000000..7a79bf110b
--- /dev/null
+++ b/src/include/fe_utils/recovery_gen.h
@@ -0,0 +1,25 @@
+/*-------------------------------------------------------------------------
+ *
+ * recovery_gen.h - general code that is used in both pg_basebackup and pg_rewind.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef RECOVERY_GEN_H
+#define RECOVERY_GEN_H
+
+#include "libpq-fe.h"
+#include "pqexpbuffer.h"
+
+extern PQExpBuffer GenerateRecoveryConfig(PGconn *pgconn, char *pg_replication_slot);
+extern void WriteRecoveryConfig(PGconn *pgconn, char *target_dir, PQExpBuffer contents);
+
+/*
+ * recovery.conf is integrated into postgresql.conf from version 12.
+ */
+#define MINIMUM_VERSION_FOR_RECOVERY_GUC 120000
+
+
+#endif /* RECOVERY_GEN_H */
--
2.17.2
v6-0003-Ensure-target-clean-shutdown-at-the-beginning-of-.patch (application/octet-stream, 5.9K) download
From 22647fbce98364fd56460120f9759bdc0d93bd99 Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Tue, 19 Mar 2019 12:41:30 +0800
Subject: [PATCH v6 3/3] Ensure target clean shutdown at the beginning of
pg_rewind
---
doc/src/sgml/ref/pg_rewind.sgml | 16 ++++++++
src/bin/pg_rewind/pg_rewind.c | 70 ++++++++++++++++++++++++++++++++-
2 files changed, 85 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 6a46b8c3b4..a7e001a864 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -176,6 +176,22 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-s</option></term>
+ <term><option>--skip-clean-shutdown</option></term>
+ <listitem>
+ <para>
+ By default, <application>pg_rewind</application> makes sure target is
+ clean shutdown before doing recovery by launching a single mode
+ postgres instance if needed to ensure the target has completed crash
+ recovery and pg_control is in clean shutdown state. Clean shutdown
+ state is a prerequisite of <application>pg_rewind</application>.
+ With this option, <application>pg_rewind</application> skips this and
+ then users need to do this themselves.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-n</option></term>
<term><option>--dry-run</option></term>
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index ddf71679b2..816f30365d 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -41,6 +41,7 @@ static void digestControlFile(ControlFileData *ControlFile, char *source,
static void syncTargetDirectory(void);
static void sanityChecks(void);
static void findCommonAncestorTimeline(XLogRecPtr *recptr, int *tliIndex);
+static void ensureCleanShutdown(const char *argv0);
static ControlFileData ControlFile_target;
static ControlFileData ControlFile_source;
@@ -77,6 +78,7 @@ usage(const char *progname)
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
+ printf(_(" -s, --skip-clean-shutdown skip running single-mode postgres if needed to make sure target is clean shutdown\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -103,6 +105,7 @@ main(int argc, char **argv)
{"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
+ {"skip-clean-shutdown", no_argument, NULL, 's'},
{"version", no_argument, NULL, 'V'},
{"dry-run", no_argument, NULL, 'n'},
{"no-sync", no_argument, NULL, 'N'},
@@ -119,6 +122,7 @@ main(int argc, char **argv)
XLogRecPtr chkptredo;
size_t size;
char *buffer;
+ bool skip_clean_shutdown = false;
bool rewind_needed;
XLogRecPtr endrec;
TimeLineID endtli;
@@ -144,7 +148,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPRs", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -152,6 +156,10 @@ main(int argc, char **argv)
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
+ case 's':
+ skip_clean_shutdown = true;
+ break;
+
case 'P':
showprogress = true;
break;
@@ -258,6 +266,22 @@ main(int argc, char **argv)
digestControlFile(&ControlFile_target, buffer, size);
pg_free(buffer);
+ /*
+ * If the target instance was not cleanly shut down, run a single-user
+ * postgres session really quickly and reload the control file to get the
+ * new state.
+ */
+ if (!skip_clean_shutdown &&
+ ControlFile_target.state != DB_SHUTDOWNED &&
+ ControlFile_target.state != DB_SHUTDOWNED_IN_RECOVERY)
+ {
+ ensureCleanShutdown(argv[0]);
+
+ buffer = slurpFile(datadir_target, "global/pg_control", &size);
+ digestControlFile(&ControlFile_target, buffer, size);
+ pg_free(buffer);
+ }
+
buffer = fetchFile("global/pg_control", &size);
digestControlFile(&ControlFile_source, buffer, size);
pg_free(buffer);
@@ -772,3 +796,47 @@ syncTargetDirectory(void)
fsync_pgdata(datadir_target, PG_VERSION_NUM);
}
+
+/*
+ * Ensure clean shutdown of target instance by launching single-user mode
+ * postgres to do crash recovery.
+ */
+static void
+ensureCleanShutdown(const char *argv0)
+{
+ int ret;
+#define MAXCMDLEN (2 * MAXPGPATH)
+ char exec_path[MAXPGPATH];
+ char cmd[MAXCMDLEN];
+
+ /* locate postgres binary */
+ if ((ret = find_other_exec(argv0, "postgres",
+ PG_BACKEND_VERSIONSTR,
+ exec_path)) < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(argv0, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+
+ if (ret == -1)
+ pg_fatal("The program \"postgres\" is needed by %s but was \n"
+ "not found in the same directory as \"%s\".\n"
+ "Check your installation.\n", progname, full_path);
+ else
+ pg_fatal("The program \"postgres\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.\n", full_path, progname);
+ }
+
+ /* only skip processing after ensuring presence of postgres */
+ if (dry_run)
+ return;
+
+ /* finally run postgres single-user mode */
+ snprintf(cmd, MAXCMDLEN, "\"%s\" --single -D \"%s\" template1 < %s",
+ exec_path, datadir_target, DEVNULL);
+
+ if (system(cmd) != 0)
+ pg_fatal("postgres single-user mode of target instance failed for command: %s\n", cmd);
+}
--
2.17.2
v6-0002-Add-option-to-write-recovery-configuration-inform.patch (application/octet-stream, 4.3K) download
From 8888da2b81a3bcb64c125c3df71deec0db6903e5 Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 19:37:36 +0800
Subject: [PATCH v6 2/3] Add option to write recovery configuration information
in pg_rewind.
---
doc/src/sgml/ref/pg_rewind.sgml | 11 +++++++++++
src/bin/pg_rewind/Makefile | 2 +-
src/bin/pg_rewind/pg_rewind.c | 17 ++++++++++++++++-
3 files changed, 28 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 52a1caa246..6a46b8c3b4 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -165,6 +165,17 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-R</option></term>
+ <term><option>--write-recovery-conf</option></term>
+ <listitem>
+ <para>
+ Create <filename>standby.signal</filename> and append connection settings
+ to <filename>postgresql.auto.conf</filename> in the output directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-n</option></term>
<term><option>--dry-run</option></term>
diff --git a/src/bin/pg_rewind/Makefile b/src/bin/pg_rewind/Makefile
index 859d3abc41..98fb381c62 100644
--- a/src/bin/pg_rewind/Makefile
+++ b/src/bin/pg_rewind/Makefile
@@ -16,7 +16,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I$(libpq_srcdir) -DFRONTEND $(CPPFLAGS)
-LDFLAGS_INTERNAL += $(libpq_pgport)
+LDFLAGS_INTERNAL += $(libpq_pgport) -L$(top_builddir)/src/fe_utils -lpgfeutils
OBJS = pg_rewind.o parsexlog.o xlogreader.o datapagemap.o timeline.o \
fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o \
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index c3903c5753..ddf71679b2 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -29,6 +29,7 @@
#include "common/restricted_token.h"
#include "getopt_long.h"
#include "storage/bufpage.h"
+#include "fe_utils/recovery_gen.h"
static void usage(const char *progname);
@@ -75,6 +76,7 @@ usage(const char *progname)
printf(_(" -D, --target-pgdata=DIRECTORY existing data directory to modify\n"));
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
+ printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -98,6 +100,7 @@ main(int argc, char **argv)
static struct option long_options[] = {
{"help", no_argument, NULL, '?'},
{"target-pgdata", required_argument, NULL, 'D'},
+ {"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
{"version", no_argument, NULL, 'V'},
@@ -120,6 +123,7 @@ main(int argc, char **argv)
XLogRecPtr endrec;
TimeLineID endtli;
ControlFileData ControlFile_new;
+ bool writerecoveryconf = false;
pg_logging_init(argv[0]);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_rewind"));
@@ -140,7 +144,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNP", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -160,6 +164,10 @@ main(int argc, char **argv)
do_sync = false;
break;
+ case 'R':
+ writerecoveryconf = true;
+ break;
+
case 3:
debug = true;
pg_logging_set_level(PG_LOG_DEBUG);
@@ -306,6 +314,9 @@ main(int argc, char **argv)
if (!rewind_needed)
{
pg_log_info("no rewind required");
+ if (writerecoveryconf && connstr_source)
+ WriteRecoveryConfig(conn, datadir_target,
+ GenerateRecoveryConfig(conn, NULL));
exit(0);
}
@@ -403,6 +414,10 @@ main(int argc, char **argv)
pg_log_info("syncing target data directory");
syncTargetDirectory();
+ if (writerecoveryconf && connstr_source)
+ WriteRecoveryConfig(conn, datadir_target,
+ GenerateRecoveryConfig(conn, NULL));
+
pg_log_info("Done!");
return 0;
--
2.17.2
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-09-20 07:33 ` Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 46+ messages in thread
From: Paul Guo @ 2019-09-20 07:33 UTC (permalink / raw)
To: Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
The patch series failed on Windows CI. I modified the Windows build file to
fix that. See attached for the v7 version.
Attachments: download all (3 files, .tar.gz)
v7-0003-Ensure-target-clean-shutdown-at-the-beginning-of-.patch (application/octet-stream, 5.9K) download
From a81a58f666b1dc1caf0241a6683f984c3cec345c Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Tue, 19 Mar 2019 12:41:30 +0800
Subject: [PATCH v7 3/3] Ensure target clean shutdown at the beginning of
pg_rewind
---
doc/src/sgml/ref/pg_rewind.sgml | 16 ++++++++
src/bin/pg_rewind/pg_rewind.c | 70 ++++++++++++++++++++++++++++++++-
2 files changed, 85 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 6a46b8c3b4..a7e001a864 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -176,6 +176,22 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-s</option></term>
+ <term><option>--skip-clean-shutdown</option></term>
+ <listitem>
+ <para>
+ By default, <application>pg_rewind</application> makes sure target is
+ clean shutdown before doing recovery by launching a single mode
+ postgres instance if needed to ensure the target has completed crash
+ recovery and pg_control is in clean shutdown state. Clean shutdown
+ state is a prerequisite of <application>pg_rewind</application>.
+ With this option, <application>pg_rewind</application> skips this and
+ then users need to do this themselves.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-n</option></term>
<term><option>--dry-run</option></term>
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index ddf71679b2..816f30365d 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -41,6 +41,7 @@ static void digestControlFile(ControlFileData *ControlFile, char *source,
static void syncTargetDirectory(void);
static void sanityChecks(void);
static void findCommonAncestorTimeline(XLogRecPtr *recptr, int *tliIndex);
+static void ensureCleanShutdown(const char *argv0);
static ControlFileData ControlFile_target;
static ControlFileData ControlFile_source;
@@ -77,6 +78,7 @@ usage(const char *progname)
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
+ printf(_(" -s, --skip-clean-shutdown skip running single-mode postgres if needed to make sure target is clean shutdown\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -103,6 +105,7 @@ main(int argc, char **argv)
{"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
+ {"skip-clean-shutdown", no_argument, NULL, 's'},
{"version", no_argument, NULL, 'V'},
{"dry-run", no_argument, NULL, 'n'},
{"no-sync", no_argument, NULL, 'N'},
@@ -119,6 +122,7 @@ main(int argc, char **argv)
XLogRecPtr chkptredo;
size_t size;
char *buffer;
+ bool skip_clean_shutdown = false;
bool rewind_needed;
XLogRecPtr endrec;
TimeLineID endtli;
@@ -144,7 +148,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPRs", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -152,6 +156,10 @@ main(int argc, char **argv)
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
+ case 's':
+ skip_clean_shutdown = true;
+ break;
+
case 'P':
showprogress = true;
break;
@@ -258,6 +266,22 @@ main(int argc, char **argv)
digestControlFile(&ControlFile_target, buffer, size);
pg_free(buffer);
+ /*
+ * If the target instance was not cleanly shut down, run a single-user
+ * postgres session really quickly and reload the control file to get the
+ * new state.
+ */
+ if (!skip_clean_shutdown &&
+ ControlFile_target.state != DB_SHUTDOWNED &&
+ ControlFile_target.state != DB_SHUTDOWNED_IN_RECOVERY)
+ {
+ ensureCleanShutdown(argv[0]);
+
+ buffer = slurpFile(datadir_target, "global/pg_control", &size);
+ digestControlFile(&ControlFile_target, buffer, size);
+ pg_free(buffer);
+ }
+
buffer = fetchFile("global/pg_control", &size);
digestControlFile(&ControlFile_source, buffer, size);
pg_free(buffer);
@@ -772,3 +796,47 @@ syncTargetDirectory(void)
fsync_pgdata(datadir_target, PG_VERSION_NUM);
}
+
+/*
+ * Ensure clean shutdown of target instance by launching single-user mode
+ * postgres to do crash recovery.
+ */
+static void
+ensureCleanShutdown(const char *argv0)
+{
+ int ret;
+#define MAXCMDLEN (2 * MAXPGPATH)
+ char exec_path[MAXPGPATH];
+ char cmd[MAXCMDLEN];
+
+ /* locate postgres binary */
+ if ((ret = find_other_exec(argv0, "postgres",
+ PG_BACKEND_VERSIONSTR,
+ exec_path)) < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(argv0, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+
+ if (ret == -1)
+ pg_fatal("The program \"postgres\" is needed by %s but was \n"
+ "not found in the same directory as \"%s\".\n"
+ "Check your installation.\n", progname, full_path);
+ else
+ pg_fatal("The program \"postgres\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.\n", full_path, progname);
+ }
+
+ /* only skip processing after ensuring presence of postgres */
+ if (dry_run)
+ return;
+
+ /* finally run postgres single-user mode */
+ snprintf(cmd, MAXCMDLEN, "\"%s\" --single -D \"%s\" template1 < %s",
+ exec_path, datadir_target, DEVNULL);
+
+ if (system(cmd) != 0)
+ pg_fatal("postgres single-user mode of target instance failed for command: %s\n", cmd);
+}
--
2.17.2
v7-0001-Extact-common-recovery-related-functions-from-pg_.patch (application/octet-stream, 14.8K) download
From a24aca9b9993129a74caa453def1126f890cae9e Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 18:55:09 +0800
Subject: [PATCH v7 1/3] Extact common recovery related functions from
pg_basebackup into separate files for additional pg_rewind use.
---
src/bin/pg_basebackup/pg_basebackup.c | 162 +------------------------
src/bin/pg_rewind/libpq_fetch.c | 3 +-
src/bin/pg_rewind/pg_rewind.c | 9 ++
src/bin/pg_rewind/pg_rewind.h | 3 +
src/fe_utils/Makefile | 2 +-
src/fe_utils/recovery_gen.c | 165 ++++++++++++++++++++++++++
src/include/fe_utils/recovery_gen.h | 25 ++++
src/tools/msvc/Mkvcbuild.pm | 2 +-
8 files changed, 208 insertions(+), 163 deletions(-)
create mode 100644 src/fe_utils/recovery_gen.c
create mode 100644 src/include/fe_utils/recovery_gen.h
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 7986872f10..55ef13926d 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -31,6 +31,7 @@
#include "common/file_utils.h"
#include "common/logging.h"
#include "common/string.h"
+#include "fe_utils/recovery_gen.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
#include "libpq-fe.h"
@@ -67,11 +68,6 @@ typedef struct TablespaceList
*/
#define MINIMUM_VERSION_FOR_TEMP_SLOTS 100000
-/*
- * recovery.conf is integrated into postgresql.conf from version 12.
- */
-#define MINIMUM_VERSION_FOR_RECOVERY_GUC 120000
-
/*
* Different ways to include WAL
*/
@@ -147,8 +143,6 @@ static void progress_report(int tablespacenum, const char *filename, bool force)
static void ReceiveTarFile(PGconn *conn, PGresult *res, int rownum);
static void ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum);
-static void GenerateRecoveryConf(PGconn *conn);
-static void WriteRecoveryConf(void);
static void BaseBackup(void);
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
@@ -1629,7 +1623,7 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
PQfreemem(copybuf);
if (basetablespace && writerecoveryconf)
- WriteRecoveryConf();
+ WriteRecoveryConfig(conn, basedir, recoveryconfcontents);
/*
* No data is synced here, everything is done for all tablespaces at the
@@ -1637,156 +1631,6 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
*/
}
-/*
- * Escape a string so that it can be used as a value in a key-value pair
- * a configuration file.
- */
-static char *
-escape_quotes(const char *src)
-{
- char *result = escape_single_quotes_ascii(src);
-
- if (!result)
- {
- pg_log_error("out of memory");
- exit(1);
- }
- return result;
-}
-
-/*
- * Create a configuration file in memory using a PQExpBuffer
- */
-static void
-GenerateRecoveryConf(PGconn *conn)
-{
- PQconninfoOption *connOptions;
- PQconninfoOption *option;
- PQExpBufferData conninfo_buf;
- char *escaped;
-
- recoveryconfcontents = createPQExpBuffer();
- if (!recoveryconfcontents)
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- /*
- * In PostgreSQL 12 and newer versions, standby_mode is gone, replaced by
- * standby.signal to trigger a standby state at recovery.
- */
- if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
- appendPQExpBufferStr(recoveryconfcontents, "standby_mode = 'on'\n");
-
- connOptions = PQconninfo(conn);
- if (connOptions == NULL)
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- initPQExpBuffer(&conninfo_buf);
- for (option = connOptions; option && option->keyword; option++)
- {
- /* Omit empty settings and those libpqwalreceiver overrides. */
- if (strcmp(option->keyword, "replication") == 0 ||
- strcmp(option->keyword, "dbname") == 0 ||
- strcmp(option->keyword, "fallback_application_name") == 0 ||
- (option->val == NULL) ||
- (option->val != NULL && option->val[0] == '\0'))
- continue;
-
- /* Separate key-value pairs with spaces */
- if (conninfo_buf.len != 0)
- appendPQExpBufferChar(&conninfo_buf, ' ');
-
- /*
- * Write "keyword=value" pieces, the value string is escaped and/or
- * quoted if necessary.
- */
- appendPQExpBuffer(&conninfo_buf, "%s=", option->keyword);
- appendConnStrVal(&conninfo_buf, option->val);
- }
-
- /*
- * Escape the connection string, so that it can be put in the config file.
- * Note that this is different from the escaping of individual connection
- * options above!
- */
- escaped = escape_quotes(conninfo_buf.data);
- appendPQExpBuffer(recoveryconfcontents, "primary_conninfo = '%s'\n", escaped);
- free(escaped);
-
- if (replication_slot)
- {
- /* unescaped: ReplicationSlotValidateName allows [a-z0-9_] only */
- appendPQExpBuffer(recoveryconfcontents, "primary_slot_name = '%s'\n",
- replication_slot);
- }
-
- if (PQExpBufferBroken(recoveryconfcontents) ||
- PQExpBufferDataBroken(conninfo_buf))
- {
- pg_log_error("out of memory");
- exit(1);
- }
-
- termPQExpBuffer(&conninfo_buf);
-
- PQconninfoFree(connOptions);
-}
-
-
-/*
- * Write the configuration file into the directory specified in basedir,
- * with the contents already collected in memory appended. Then write
- * the signal file into the basedir. If the server does not support
- * recovery parameters as GUCs, the signal file is not necessary, and
- * configuration is written to recovery.conf.
- */
-static void
-WriteRecoveryConf(void)
-{
- char filename[MAXPGPATH];
- FILE *cf;
- bool is_recovery_guc_supported = true;
-
- if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
- is_recovery_guc_supported = false;
-
- snprintf(filename, MAXPGPATH, "%s/%s", basedir,
- is_recovery_guc_supported ? "postgresql.auto.conf" : "recovery.conf");
-
- cf = fopen(filename, is_recovery_guc_supported ? "a" : "w");
- if (cf == NULL)
- {
- pg_log_error("could not open file \"%s\": %m", filename);
- exit(1);
- }
-
- if (fwrite(recoveryconfcontents->data, recoveryconfcontents->len, 1, cf) != 1)
- {
- pg_log_error("could not write to file \"%s\": %m", filename);
- exit(1);
- }
-
- fclose(cf);
-
- if (is_recovery_guc_supported)
- {
- snprintf(filename, MAXPGPATH, "%s/%s", basedir, "standby.signal");
- cf = fopen(filename, "w");
- if (cf == NULL)
- {
- pg_log_error("could not create file \"%s\": %m", filename);
- exit(1);
- }
-
- fclose(cf);
- }
-}
-
static void
BaseBackup(void)
@@ -1843,7 +1687,7 @@ BaseBackup(void)
* Build contents of configuration file if requested
*/
if (writerecoveryconf)
- GenerateRecoveryConf(conn);
+ recoveryconfcontents = GenerateRecoveryConfig(conn, replication_slot);
/*
* Run IDENTIFY_SYSTEM so we can get the timeline
diff --git a/src/bin/pg_rewind/libpq_fetch.c b/src/bin/pg_rewind/libpq_fetch.c
index 002776f696..f4ebf7d842 100644
--- a/src/bin/pg_rewind/libpq_fetch.c
+++ b/src/bin/pg_rewind/libpq_fetch.c
@@ -20,12 +20,11 @@
#include "file_ops.h"
#include "filemap.h"
-#include "libpq-fe.h"
#include "catalog/pg_type_d.h"
#include "fe_utils/connect.h"
#include "port/pg_bswap.h"
-static PGconn *conn = NULL;
+PGconn *conn = NULL;
/*
* Files are fetched max CHUNKSIZE bytes at a time.
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index 15e3eab550..c3903c5753 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -85,6 +85,12 @@ usage(const char *progname)
printf(_("\nReport bugs to <pgsql-bugs@lists.postgresql.org>.\n"));
}
+static void
+disconnect_atexit(void)
+{
+ if (conn != NULL)
+ PQfinish(conn);
+}
int
main(int argc, char **argv)
@@ -231,7 +237,10 @@ main(int argc, char **argv)
/* Connect to remote server */
if (connstr_source)
+ {
libpqConnect(connstr_source);
+ atexit(disconnect_atexit);
+ }
/*
* Ok, we have all the options and we're ready to start. Read in all the
diff --git a/src/bin/pg_rewind/pg_rewind.h b/src/bin/pg_rewind/pg_rewind.h
index 1125c7e60f..d23c855107 100644
--- a/src/bin/pg_rewind/pg_rewind.h
+++ b/src/bin/pg_rewind/pg_rewind.h
@@ -19,6 +19,8 @@
#include "common/logging.h"
+#include "libpq-fe.h"
+
/* Configuration options */
extern char *datadir_target;
extern char *datadir_source;
@@ -26,6 +28,7 @@ extern char *connstr_source;
extern bool showprogress;
extern bool dry_run;
extern int WalSegSz;
+extern PGconn *conn;
/* Target history */
extern TimeLineHistoryEntry *targetHistory;
diff --git a/src/fe_utils/Makefile b/src/fe_utils/Makefile
index 7d73800323..ae0f6c884b 100644
--- a/src/fe_utils/Makefile
+++ b/src/fe_utils/Makefile
@@ -19,7 +19,7 @@ include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -DFRONTEND -I$(libpq_srcdir) $(CPPFLAGS)
-OBJS = mbprint.o print.o psqlscan.o simple_list.o string_utils.o conditional.o
+OBJS = mbprint.o print.o psqlscan.o simple_list.o string_utils.o conditional.o recovery_gen.o
all: libpgfeutils.a
diff --git a/src/fe_utils/recovery_gen.c b/src/fe_utils/recovery_gen.c
new file mode 100644
index 0000000000..4e0f6a154a
--- /dev/null
+++ b/src/fe_utils/recovery_gen.c
@@ -0,0 +1,165 @@
+/*-------------------------------------------------------------------------
+ *
+ * recovery_gen.c - general code that is used in both pg_basebackup and pg_rewind.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+#include "common/logging.h"
+#include "fe_utils/string_utils.h"
+#include "fe_utils/recovery_gen.h"
+
+/*
+ * Escape a string so that it can be used as a value in a key-value pair
+ * a configuration file.
+ */
+static char *
+escape_quotes(const char *src)
+{
+ char *result = escape_single_quotes_ascii(src);
+
+ if (!result)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+ return result;
+}
+
+/*
+ * Create recovery configuration contents using a PQExpBuffer
+ */
+PQExpBuffer
+GenerateRecoveryConfig(PGconn *pgconn, char *pg_replication_slot)
+{
+ PQconninfoOption *connOptions;
+ PQconninfoOption *option;
+ PQExpBufferData conninfo_buf;
+ char *escaped;
+ PQExpBuffer contents;
+
+ contents = createPQExpBuffer();
+ if (!contents)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ /*
+ * In PostgreSQL 12 and newer versions, standby_mode is gone, replaced by
+ * standby.signal to trigger a standby state at recovery.
+ */
+ if (pgconn && PQserverVersion(pgconn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
+ appendPQExpBufferStr(contents, "standby_mode = 'on'\n");
+
+ connOptions = PQconninfo(pgconn);
+ if (connOptions == NULL)
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ initPQExpBuffer(&conninfo_buf);
+ for (option = connOptions; option && option->keyword; option++)
+ {
+ /* Omit empty settings and those libpqwalreceiver overrides. */
+ if (strcmp(option->keyword, "replication") == 0 ||
+ strcmp(option->keyword, "dbname") == 0 ||
+ strcmp(option->keyword, "fallback_application_name") == 0 ||
+ (option->val == NULL) ||
+ (option->val != NULL && option->val[0] == '\0'))
+ continue;
+
+ /* Separate key-value pairs with spaces */
+ if (conninfo_buf.len != 0)
+ appendPQExpBufferChar(&conninfo_buf, ' ');
+
+ /*
+ * Write "keyword=value" pieces, the value string is escaped and/or
+ * quoted if necessary.
+ */
+ appendPQExpBuffer(&conninfo_buf, "%s=", option->keyword);
+ appendConnStrVal(&conninfo_buf, option->val);
+ }
+
+ /*
+ * Escape the connection string, so that it can be put in the config file.
+ * Note that this is different from the escaping of individual connection
+ * options above!
+ */
+ escaped = escape_quotes(conninfo_buf.data);
+ appendPQExpBuffer(contents, "primary_conninfo = '%s'\n", escaped);
+ free(escaped);
+
+ if (pg_replication_slot)
+ {
+ /* unescaped: ReplicationSlotValidateName allows [a-z0-9_] only */
+ appendPQExpBuffer(contents, "primary_slot_name = '%s'\n",
+ pg_replication_slot);
+ }
+
+ if (PQExpBufferBroken(contents) ||
+ PQExpBufferDataBroken(conninfo_buf))
+ {
+ pg_log_error("out of memory");
+ exit(1);
+ }
+
+ termPQExpBuffer(&conninfo_buf);
+
+ PQconninfoFree(connOptions);
+
+ return contents;
+}
+
+/*
+ * Write the configuration file in the directory specified in target_dir,
+ * with the contents already collected in memory appended. Then write
+ * the signal file into the target_dir. If the server does not support
+ * recovery parameters as GUCs, the signal file is not necessary, and
+ * configuration is written to recovery.conf.
+ */
+void
+WriteRecoveryConfig(PGconn *pgconn, char *target_dir, PQExpBuffer contents)
+{
+ char filename[MAXPGPATH];
+ FILE *cf;
+ bool is_recovery_guc_supported = true;
+
+ if (pgconn && PQserverVersion(pgconn) < MINIMUM_VERSION_FOR_RECOVERY_GUC)
+ is_recovery_guc_supported = false;
+
+ snprintf(filename, MAXPGPATH, "%s/%s", target_dir,
+ is_recovery_guc_supported ? "postgresql.auto.conf" : "recovery.conf");
+
+ cf = fopen(filename, is_recovery_guc_supported ? "a" : "w");
+ if (cf == NULL)
+ {
+ pg_log_error("could not open file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ if (fwrite(contents->data, contents->len, 1, cf) != 1)
+ {
+ pg_log_error("could not write to file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ fclose(cf);
+
+ if (is_recovery_guc_supported)
+ {
+ snprintf(filename, MAXPGPATH, "%s/%s", target_dir, "standby.signal");
+ cf = fopen(filename, "w");
+ if (cf == NULL)
+ {
+ pg_log_error("could not create file \"%s\": %m", filename);
+ exit(1);
+ }
+
+ fclose(cf);
+ }
+}
diff --git a/src/include/fe_utils/recovery_gen.h b/src/include/fe_utils/recovery_gen.h
new file mode 100644
index 0000000000..7a79bf110b
--- /dev/null
+++ b/src/include/fe_utils/recovery_gen.h
@@ -0,0 +1,25 @@
+/*-------------------------------------------------------------------------
+ *
+ * recovery_gen.h - general code that is used in both pg_basebackup and pg_rewind.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef RECOVERY_GEN_H
+#define RECOVERY_GEN_H
+
+#include "libpq-fe.h"
+#include "pqexpbuffer.h"
+
+extern PQExpBuffer GenerateRecoveryConfig(PGconn *pgconn, char *pg_replication_slot);
+extern void WriteRecoveryConfig(PGconn *pgconn, char *target_dir, PQExpBuffer contents);
+
+/*
+ * recovery.conf is integrated into postgresql.conf from version 12.
+ */
+#define MINIMUM_VERSION_FOR_RECOVERY_GUC 120000
+
+
+#endif /* RECOVERY_GEN_H */
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 239f13cc12..cd6bb7eef6 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -141,7 +141,7 @@ sub mkvcbuild
our @pgcommonbkndfiles = @pgcommonallfiles;
our @pgfeutilsfiles = qw(
- conditional.c mbprint.c print.c psqlscan.l psqlscan.c simple_list.c string_utils.c);
+ conditional.c mbprint.c print.c psqlscan.l psqlscan.c simple_list.c string_utils.c recovery_gen.c);
$libpgport = $solution->AddProject('libpgport', 'lib', 'misc');
$libpgport->AddDefine('FRONTEND');
--
2.17.2
v7-0002-Add-option-to-write-recovery-configuration-inform.patch (application/octet-stream, 4.3K) download
From 8897e28e49a86996e1f9d6a3ec9740f2b7f20203 Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 19:37:36 +0800
Subject: [PATCH v7 2/3] Add option to write recovery configuration information
in pg_rewind.
---
doc/src/sgml/ref/pg_rewind.sgml | 11 +++++++++++
src/bin/pg_rewind/Makefile | 2 +-
src/bin/pg_rewind/pg_rewind.c | 17 ++++++++++++++++-
3 files changed, 28 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 52a1caa246..6a46b8c3b4 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -165,6 +165,17 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-R</option></term>
+ <term><option>--write-recovery-conf</option></term>
+ <listitem>
+ <para>
+ Create <filename>standby.signal</filename> and append connection settings
+ to <filename>postgresql.auto.conf</filename> in the output directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-n</option></term>
<term><option>--dry-run</option></term>
diff --git a/src/bin/pg_rewind/Makefile b/src/bin/pg_rewind/Makefile
index 859d3abc41..98fb381c62 100644
--- a/src/bin/pg_rewind/Makefile
+++ b/src/bin/pg_rewind/Makefile
@@ -16,7 +16,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I$(libpq_srcdir) -DFRONTEND $(CPPFLAGS)
-LDFLAGS_INTERNAL += $(libpq_pgport)
+LDFLAGS_INTERNAL += $(libpq_pgport) -L$(top_builddir)/src/fe_utils -lpgfeutils
OBJS = pg_rewind.o parsexlog.o xlogreader.o datapagemap.o timeline.o \
fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o \
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index c3903c5753..ddf71679b2 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -29,6 +29,7 @@
#include "common/restricted_token.h"
#include "getopt_long.h"
#include "storage/bufpage.h"
+#include "fe_utils/recovery_gen.h"
static void usage(const char *progname);
@@ -75,6 +76,7 @@ usage(const char *progname)
printf(_(" -D, --target-pgdata=DIRECTORY existing data directory to modify\n"));
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
+ printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -98,6 +100,7 @@ main(int argc, char **argv)
static struct option long_options[] = {
{"help", no_argument, NULL, '?'},
{"target-pgdata", required_argument, NULL, 'D'},
+ {"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
{"version", no_argument, NULL, 'V'},
@@ -120,6 +123,7 @@ main(int argc, char **argv)
XLogRecPtr endrec;
TimeLineID endtli;
ControlFileData ControlFile_new;
+ bool writerecoveryconf = false;
pg_logging_init(argv[0]);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_rewind"));
@@ -140,7 +144,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNP", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -160,6 +164,10 @@ main(int argc, char **argv)
do_sync = false;
break;
+ case 'R':
+ writerecoveryconf = true;
+ break;
+
case 3:
debug = true;
pg_logging_set_level(PG_LOG_DEBUG);
@@ -306,6 +314,9 @@ main(int argc, char **argv)
if (!rewind_needed)
{
pg_log_info("no rewind required");
+ if (writerecoveryconf && connstr_source)
+ WriteRecoveryConfig(conn, datadir_target,
+ GenerateRecoveryConfig(conn, NULL));
exit(0);
}
@@ -403,6 +414,10 @@ main(int argc, char **argv)
pg_log_info("syncing target data directory");
syncTargetDirectory();
+ if (writerecoveryconf && connstr_source)
+ WriteRecoveryConfig(conn, datadir_target,
+ GenerateRecoveryConfig(conn, NULL));
+
pg_log_info("Done!");
return 0;
--
2.17.2
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-09-24 20:12 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
0 siblings, 1 reply; 46+ messages in thread
From: Alvaro Herrera @ 2019-09-24 20:12 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On 2019-Sep-20, Paul Guo wrote:
> The patch series failed on Windows CI. I modified the Windows build file to
> fix that. See attached for the v7 version.
Thanks.
Question about 0003. If we specify --skip-clean-shutdown and the cluter
was not cleanly shut down, shouldn't we error out instead of trying to
press on?
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2019-09-25 02:03 ` Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 46+ messages in thread
From: Paul Guo @ 2019-09-25 02:03 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
>
> On 2019-Sep-20, Paul Guo wrote:
>
> > The patch series failed on Windows CI. I modified the Windows build file
> to
> > fix that. See attached for the v7 version.
>
> Thanks.
>
> Question about 0003. If we specify --skip-clean-shutdown and the cluter
> was not cleanly shut down, shouldn't we error out instead of trying to
> press on?
pg_rewind would error out in this case, see sanityChecks().
Users are expected to clean up themselves if they use this argument.
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-09-25 17:48 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 19:26 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Laurenz Albe <laurenz.albe@cybertec.at>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
0 siblings, 2 replies; 46+ messages in thread
From: Alvaro Herrera @ 2019-09-25 17:48 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>; Alexey Kondratov <a.kondratov@postgrespro.ru>
CC Alexey for reasons that become clear below.
On 2019-Sep-25, Paul Guo wrote:
> > Question about 0003. If we specify --skip-clean-shutdown and the cluter
> > was not cleanly shut down, shouldn't we error out instead of trying to
> > press on?
>
> pg_rewind would error out in this case, see sanityChecks().
> Users are expected to clean up themselves if they use this argument.
Ah, good. We should have a comment about that below the relevant
stanza, I suggest. (Or maybe in the same comment that ends in line
272).
I pushed 0001 with a few tweaks. Nothing really substantial, just my
OCD that doesn't leave me alone ... but this means your subsequent
patches need to be adjusted. One thing is that that patch touched
pg_rewind for no reason (those changes should have been in 0002) --
dropped those.
Another thing in 0002 is that you're adding a "-R" switch to pg_rewind,
but we have another patch in the commitfest using the same switch for a
different purpose. Maybe you guys need to get to an agreement over who
uses the letter :-) Also, it would be super helpful if you review
Alexey's patch: https://commitfest.postgresql.org/24/1849/
This line is far too long:
+ printf(_(" -s, --skip-clean-shutdown skip running single-mode postgres if needed to make sure target is clean shutdown\n"));
Can we make the description more concise?
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2019-09-25 19:26 ` Laurenz Albe <laurenz.albe@cybertec.at>
2019-09-25 20:22 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) a.kondratov@postgrespro.ru
1 sibling, 1 reply; 46+ messages in thread
From: Laurenz Albe @ 2019-09-25 19:26 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; Paul Guo <pguo@pivotal.io>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>; Alexey Kondratov <a.kondratov@postgrespro.ru>
On Wed, 2019-09-25 at 14:48 -0300, Alvaro Herrera wrote:
> Another thing in 0002 is that you're adding a "-R" switch to
> pg_rewind, but we have another patch in the commitfest using the same
> switch for a different purpose. Maybe you guys need to get to an
> agreement over who uses the letter :-) Also, it would be super
> helpful if you review Alexey's patch:
> https://commitfest.postgresql.org/24/1849/
I believe that -R should be reserved for creating recovery.conf,
similar to pg_basebackup.
Everything else would be confusing.
I've been missing pg_rewind -R!
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 19:26 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Laurenz Albe <laurenz.albe@cybertec.at>
@ 2019-09-25 20:22 ` a.kondratov@postgrespro.ru
2019-09-26 14:09 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
0 siblings, 1 reply; 46+ messages in thread
From: a.kondratov@postgrespro.ru @ 2019-09-25 20:22 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Paul Guo <pguo@pivotal.io>; Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On 2019-09-25 20:48, Alvaro Herrera wrote:
> CC Alexey for reasons that become clear below.
>
> Another thing in 0002 is that you're adding a "-R" switch to pg_rewind,
> but we have another patch in the commitfest using the same switch for a
> different purpose. Maybe you guys need to get to an agreement over who
> uses the letter :-)
>
Thank you for mentioning me. I've been monitoring silently this thread
and was ready to modify my patch if this one will proceed faster. It
seems like it's time :)
On 2019-09-25 22:26, Laurenz Albe wrote:
>
> I believe that -R should be reserved for creating recovery.conf,
> similar to pg_basebackup.
>
> Everything else would be confusing.
>
> I've been missing pg_rewind -R!
>
Yes, -R is already used in pg_basebackup for the same functionality, so
it seems natural to use it here as well for consistency.
I will review options naming in my own patch and update it accordingly.
Maybe -w/-W or -a/-A options will be good, since it's about WALs
retrieval from archive.
Regards
--
Alexey
P.S. Just noticed that in v12 fullname of -R option in pg_basebackup is
still --write-recovery-conf, which is good for a backward compatibility,
but looks a little bit awkward, since recovery.conf doesn't exist
already, doesn't it? However, one may read it as
'write-recovery-configuration', then it seems fine.
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 19:26 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Laurenz Albe <laurenz.albe@cybertec.at>
2019-09-25 20:22 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) a.kondratov@postgrespro.ru
@ 2019-09-26 14:09 ` Paul Guo <pguo@pivotal.io>
0 siblings, 0 replies; 46+ messages in thread
From: Paul Guo @ 2019-09-26 14:09 UTC (permalink / raw)
To: a.kondratov@postgrespro.ru; +Cc: Laurenz Albe <laurenz.albe@cybertec.at>; Alvaro Herrera <alvherre@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
>
>
> Yes, -R is already used in pg_basebackup for the same functionality, so
> it seems natural to use it here as well for consistency.
>
> I will review options naming in my own patch and update it accordingly.
> Maybe -w/-W or -a/-A options will be good, since it's about WALs
> retrieval from archive.
>
>
Thanks
>
> Regards
> --
> Alexey
>
> P.S. Just noticed that in v12 fullname of -R option in pg_basebackup is
> still --write-recovery-conf, which is good for a backward compatibility,
> but looks a little bit awkward, since recovery.conf doesn't exist
> already, doesn't it? However, one may read it as
> 'write-recovery-configuration', then it seems fine.
>
>
Yes, here is the description
"--write-recovery-conf write configuration for replication"
So we do not mention that is the file recovery.conf. People do not know
about the recovery.conf history might really not be confused since
postgresql has various configuration files.
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2019-09-26 14:05 ` Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 1 reply; 46+ messages in thread
From: Paul Guo @ 2019-09-26 14:05 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>; Alexey Kondratov <a.kondratov@postgrespro.ru>
On Thu, Sep 26, 2019 at 1:48 AM Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:
> CC Alexey for reasons that become clear below.
>
> On 2019-Sep-25, Paul Guo wrote:
>
> > > Question about 0003. If we specify --skip-clean-shutdown and the
> cluter
> > > was not cleanly shut down, shouldn't we error out instead of trying to
> > > press on?
> >
> > pg_rewind would error out in this case, see sanityChecks().
> > Users are expected to clean up themselves if they use this argument.
>
> Ah, good. We should have a comment about that below the relevant
> stanza, I suggest. (Or maybe in the same comment that ends in line
> 272).
>
> I pushed 0001 with a few tweaks. Nothing really substantial, just my
> OCD that doesn't leave me alone ... but this means your subsequent
> patches need to be adjusted. One thing is that that patch touched
> pg_rewind for no reason (those changes should have been in 0002) --
> dropped those.
>
> Another thing in 0002 is that you're adding a "-R" switch to pg_rewind,
> but we have another patch in the commitfest using the same switch for a
> different purpose. Maybe you guys need to get to an agreement over who
> uses the letter :-) Also, it would be super helpful if you review
> Alexey's patch: https://commitfest.postgresql.org/24/1849/
>
>
> This line is far too long:
>
> + printf(_(" -s, --skip-clean-shutdown skip running single-mode
> postgres if needed to make sure target is clean shutdown\n"));
>
> Can we make the description more concise?
>
Thanks. I've updated the reset two patches and attached as v8.
Note in the 2nd patch, the long option is changed as below. Both the option
and description
now seems to be more concise since we want db state as either DB_SHUTDOWNED
or
DB_SHUTDOWNED_IN_RECOVERY.
"-s, --no-ensure-shutdowned do not auto-fix unclean shutdown"
Attachments: download all (2 files, .tar.gz)
v8-0001-Add-option-to-write-recovery-configuration-inform.patch (application/octet-stream, 5.9K) download
From 0ea6b6322e8eb6e105936e18f433b0454dab37bd Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 19:37:36 +0800
Subject: [PATCH v8 1/2] Add option to write recovery configuration information
in pg_rewind.
---
doc/src/sgml/ref/pg_rewind.sgml | 11 +++++++++++
src/bin/pg_rewind/Makefile | 2 +-
src/bin/pg_rewind/libpq_fetch.c | 3 +--
src/bin/pg_rewind/pg_rewind.c | 26 +++++++++++++++++++++++++-
src/bin/pg_rewind/pg_rewind.h | 2 ++
5 files changed, 40 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index ac142d22fc..b8b43e8220 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -165,6 +165,17 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-R</option></term>
+ <term><option>--write-recovery-conf</option></term>
+ <listitem>
+ <para>
+ Create <filename>standby.signal</filename> and append connection settings
+ to <filename>postgresql.auto.conf</filename> in the output directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-n</option></term>
<term><option>--dry-run</option></term>
diff --git a/src/bin/pg_rewind/Makefile b/src/bin/pg_rewind/Makefile
index 859d3abc41..98fb381c62 100644
--- a/src/bin/pg_rewind/Makefile
+++ b/src/bin/pg_rewind/Makefile
@@ -16,7 +16,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I$(libpq_srcdir) -DFRONTEND $(CPPFLAGS)
-LDFLAGS_INTERNAL += $(libpq_pgport)
+LDFLAGS_INTERNAL += $(libpq_pgport) -L$(top_builddir)/src/fe_utils -lpgfeutils
OBJS = pg_rewind.o parsexlog.o xlogreader.o datapagemap.o timeline.o \
fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o \
diff --git a/src/bin/pg_rewind/libpq_fetch.c b/src/bin/pg_rewind/libpq_fetch.c
index 002776f696..f4ebf7d842 100644
--- a/src/bin/pg_rewind/libpq_fetch.c
+++ b/src/bin/pg_rewind/libpq_fetch.c
@@ -20,12 +20,11 @@
#include "file_ops.h"
#include "filemap.h"
-#include "libpq-fe.h"
#include "catalog/pg_type_d.h"
#include "fe_utils/connect.h"
#include "port/pg_bswap.h"
-static PGconn *conn = NULL;
+PGconn *conn = NULL;
/*
* Files are fetched max CHUNKSIZE bytes at a time.
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index 15e3eab550..2312e564f6 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -29,6 +29,7 @@
#include "common/restricted_token.h"
#include "getopt_long.h"
#include "storage/bufpage.h"
+#include "fe_utils/recovery_gen.h"
static void usage(const char *progname);
@@ -65,6 +66,12 @@ int targetNentries;
uint64 fetch_size;
uint64 fetch_done;
+static void
+disconnect_atexit(void)
+{
+ if (conn != NULL)
+ PQfinish(conn);
+}
static void
usage(const char *progname)
@@ -75,6 +82,7 @@ usage(const char *progname)
printf(_(" -D, --target-pgdata=DIRECTORY existing data directory to modify\n"));
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
+ printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -92,6 +100,7 @@ main(int argc, char **argv)
static struct option long_options[] = {
{"help", no_argument, NULL, '?'},
{"target-pgdata", required_argument, NULL, 'D'},
+ {"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
{"version", no_argument, NULL, 'V'},
@@ -114,6 +123,7 @@ main(int argc, char **argv)
XLogRecPtr endrec;
TimeLineID endtli;
ControlFileData ControlFile_new;
+ bool writerecoveryconf = false;
pg_logging_init(argv[0]);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_rewind"));
@@ -134,7 +144,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNP", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -154,6 +164,10 @@ main(int argc, char **argv)
do_sync = false;
break;
+ case 'R':
+ writerecoveryconf = true;
+ break;
+
case 3:
debug = true;
pg_logging_set_level(PG_LOG_DEBUG);
@@ -231,7 +245,10 @@ main(int argc, char **argv)
/* Connect to remote server */
if (connstr_source)
+ {
libpqConnect(connstr_source);
+ atexit(disconnect_atexit);
+ }
/*
* Ok, we have all the options and we're ready to start. Read in all the
@@ -297,6 +314,9 @@ main(int argc, char **argv)
if (!rewind_needed)
{
pg_log_info("no rewind required");
+ if (writerecoveryconf && connstr_source)
+ WriteRecoveryConfig(conn, datadir_target,
+ GenerateRecoveryConfig(conn, NULL));
exit(0);
}
@@ -394,6 +414,10 @@ main(int argc, char **argv)
pg_log_info("syncing target data directory");
syncTargetDirectory();
+ if (writerecoveryconf && connstr_source)
+ WriteRecoveryConfig(conn, datadir_target,
+ GenerateRecoveryConfig(conn, NULL));
+
pg_log_info("Done!");
return 0;
diff --git a/src/bin/pg_rewind/pg_rewind.h b/src/bin/pg_rewind/pg_rewind.h
index 1125c7e60f..3cee48556c 100644
--- a/src/bin/pg_rewind/pg_rewind.h
+++ b/src/bin/pg_rewind/pg_rewind.h
@@ -18,6 +18,7 @@
#include "storage/relfilenode.h"
#include "common/logging.h"
+#include "libpq-fe.h"
/* Configuration options */
extern char *datadir_target;
@@ -26,6 +27,7 @@ extern char *connstr_source;
extern bool showprogress;
extern bool dry_run;
extern int WalSegSz;
+extern PGconn *conn;
/* Target history */
extern TimeLineHistoryEntry *targetHistory;
--
2.17.2
v8-0002-Ensure-target-clean-shutdown-at-the-beginning-of-.patch (application/octet-stream, 6.0K) download
From 92b3c9ff64f1e6d99e58fb0fe9002fa5d34619a3 Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Tue, 19 Mar 2019 12:41:30 +0800
Subject: [PATCH v8 2/2] Ensure target clean shutdown at the beginning of
pg_rewind
---
doc/src/sgml/ref/pg_rewind.sgml | 16 ++++++++
src/bin/pg_rewind/pg_rewind.c | 72 ++++++++++++++++++++++++++++++++-
2 files changed, 87 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index b8b43e8220..122f792bff 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -176,6 +176,22 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-s</option></term>
+ <term><option>--no-ensure-shutdowned</option></term>
+ <listitem>
+ <para>
+ By default, <application>pg_rewind</application> makes sure target is
+ cleanly shutdown before doing recovery by launching a single mode
+ postgres instance if needed to ensure the target has completed crash
+ recovery and pg_control is in clean shutdown state. Clean shutdown
+ state is a prerequisite of <application>pg_rewind</application>.
+ With this option, <application>pg_rewind</application> skips this and
+ then users need to do this themselves.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-n</option></term>
<term><option>--dry-run</option></term>
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index 2312e564f6..342a072c18 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -41,6 +41,7 @@ static void digestControlFile(ControlFileData *ControlFile, char *source,
static void syncTargetDirectory(void);
static void sanityChecks(void);
static void findCommonAncestorTimeline(XLogRecPtr *recptr, int *tliIndex);
+static void ensureCleanShutdown(const char *argv0);
static ControlFileData ControlFile_target;
static ControlFileData ControlFile_source;
@@ -83,6 +84,7 @@ usage(const char *progname)
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
+ printf(_(" -s, --no-ensure-shutdowned do not auto-fix unclean shutdown\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -103,6 +105,7 @@ main(int argc, char **argv)
{"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
+ {"no-ensure-shutdowned", no_argument, NULL, 's'},
{"version", no_argument, NULL, 'V'},
{"dry-run", no_argument, NULL, 'n'},
{"no-sync", no_argument, NULL, 'N'},
@@ -119,6 +122,7 @@ main(int argc, char **argv)
XLogRecPtr chkptredo;
size_t size;
char *buffer;
+ bool no_ensure_shutdowned = false;
bool rewind_needed;
XLogRecPtr endrec;
TimeLineID endtli;
@@ -144,7 +148,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPRs", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -152,6 +156,10 @@ main(int argc, char **argv)
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
+ case 's':
+ no_ensure_shutdowned = true;
+ break;
+
case 'P':
showprogress = true;
break;
@@ -258,6 +266,24 @@ main(int argc, char **argv)
digestControlFile(&ControlFile_target, buffer, size);
pg_free(buffer);
+ /*
+ * If the target instance was not cleanly shut down, run a single-user
+ * postgres session really quickly and reload the control file to get the
+ * new state. Note if no_ensure_shutdowned is specified, pg_rewind won't
+ * do that automatically. That means users need to do themselves in
+ * advance, else pg_rewind will soon quit, see sanityChecks().
+ */
+ if (!no_ensure_shutdowned &&
+ ControlFile_target.state != DB_SHUTDOWNED &&
+ ControlFile_target.state != DB_SHUTDOWNED_IN_RECOVERY)
+ {
+ ensureCleanShutdown(argv[0]);
+
+ buffer = slurpFile(datadir_target, "global/pg_control", &size);
+ digestControlFile(&ControlFile_target, buffer, size);
+ pg_free(buffer);
+ }
+
buffer = fetchFile("global/pg_control", &size);
digestControlFile(&ControlFile_source, buffer, size);
pg_free(buffer);
@@ -772,3 +798,47 @@ syncTargetDirectory(void)
fsync_pgdata(datadir_target, PG_VERSION_NUM);
}
+
+/*
+ * Ensure clean shutdown of target instance by launching single-user mode
+ * postgres to do crash recovery.
+ */
+static void
+ensureCleanShutdown(const char *argv0)
+{
+ int ret;
+#define MAXCMDLEN (2 * MAXPGPATH)
+ char exec_path[MAXPGPATH];
+ char cmd[MAXCMDLEN];
+
+ /* locate postgres binary */
+ if ((ret = find_other_exec(argv0, "postgres",
+ PG_BACKEND_VERSIONSTR,
+ exec_path)) < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(argv0, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+
+ if (ret == -1)
+ pg_fatal("The program \"postgres\" is needed by %s but was \n"
+ "not found in the same directory as \"%s\".\n"
+ "Check your installation.\n", progname, full_path);
+ else
+ pg_fatal("The program \"postgres\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.\n", full_path, progname);
+ }
+
+ /* only skip processing after ensuring presence of postgres */
+ if (dry_run)
+ return;
+
+ /* finally run postgres single-user mode */
+ snprintf(cmd, MAXCMDLEN, "\"%s\" --single -D \"%s\" template1 < %s",
+ exec_path, datadir_target, DEVNULL);
+
+ if (system(cmd) != 0)
+ pg_fatal("postgres single-user mode of target instance failed for command: %s\n", cmd);
+}
--
2.17.2
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-09-26 14:51 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
0 siblings, 1 reply; 46+ messages in thread
From: Alvaro Herrera @ 2019-09-26 14:51 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>; Alexey Kondratov <a.kondratov@postgrespro.ru>
> Thanks. I've updated the reset two patches and attached as v8.
Great, thanks.
> Note in the 2nd patch, the long option is changed as below. Both the option
> and description
> now seems to be more concise since we want db state as either DB_SHUTDOWNED
> or
> DB_SHUTDOWNED_IN_RECOVERY.
>
> "-s, --no-ensure-shutdowned do not auto-fix unclean shutdown"
Note that "shutdowned" is incorrect English; we've let
it live in the code because it's not user-visible, but we should
certainly not immortalize it where it becomes so. I suppose
"--no-ensure-shutdown" is okay, although I think some may prefer
"--no-ensure-shut-down". Opinions from native speakers would be
welcome. Also, let's expand "auto-fix" to "automatically fix" (or
"repair" if there's room in the line? Not sure. Can be bikeshedded to
death I guess.)
Secondarily, I see no reason to test connstr_source rather than just
"conn" in the other patch; doing it the other way is more natural, since
it's that thing that's tested as an argument.
pg_rewind.c: Please put the new #include line keeping the alphabetical
order.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2019-09-27 03:27 ` Paul Guo <pguo@pivotal.io>
2019-09-27 12:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-27 19:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 2 replies; 46+ messages in thread
From: Paul Guo @ 2019-09-27 03:27 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>; Alexey Kondratov <a.kondratov@postgrespro.ru>
>
>
> > Note in the 2nd patch, the long option is changed as below. Both the
> option
> > and description
> > now seems to be more concise since we want db state as either
> DB_SHUTDOWNED
> > or
> > DB_SHUTDOWNED_IN_RECOVERY.
> >
> > "-s, --no-ensure-shutdowned do not auto-fix unclean shutdown"
>
> Note that "shutdowned" is incorrect English; we've let
> it live in the code because it's not user-visible, but we should
> certainly not immortalize it where it becomes so. I suppose
> "--no-ensure-shutdown" is okay, although I think some may prefer
> "--no-ensure-shut-down". Opinions from native speakers would be
> welcome. Also, let's expand "auto-fix" to "automatically fix" (or
> "repair" if there's room in the line? Not sure. Can be bikeshedded to
> death I guess.)
>
I choose that one from the below tree.
--no-ensure-shutdown
--no-ensure-shutdowned
--no-ensure-clean-shutdown
Now I agree for user experience we should not use the 2nd one. For
--no-ensure-clean-shutdown or -no-ensure-shut-down, seems too many -.
I'm using --no-ensure-shutdown in the new version unless there are better
suggestions.
>
> Secondarily, I see no reason to test connstr_source rather than just
> "conn" in the other patch; doing it the other way is more natural, since
> it's that thing that's tested as an argument.
>
> pg_rewind.c: Please put the new #include line keeping the alphabetical
> order.
>
Agreed to the above suggestions. I attached the v9.
Thanks.
Attachments: download all (2 files, .tar.gz)
v9-0001-Add-option-to-write-recovery-configuration-inform.patch (application/octet-stream, 6.0K) download
From 1fd0c52d5a8a63206e43f14768361cd24a484d41 Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 19:37:36 +0800
Subject: [PATCH v9 1/2] Add option to write recovery configuration information
in pg_rewind.
---
doc/src/sgml/ref/pg_rewind.sgml | 11 +++++++++++
src/bin/pg_rewind/Makefile | 2 +-
src/bin/pg_rewind/libpq_fetch.c | 3 +--
src/bin/pg_rewind/pg_rewind.c | 25 ++++++++++++++++++++++++-
src/bin/pg_rewind/pg_rewind.h | 4 +++-
5 files changed, 40 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index ac142d22fc..b8b43e8220 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -165,6 +165,17 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-R</option></term>
+ <term><option>--write-recovery-conf</option></term>
+ <listitem>
+ <para>
+ Create <filename>standby.signal</filename> and append connection settings
+ to <filename>postgresql.auto.conf</filename> in the output directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-n</option></term>
<term><option>--dry-run</option></term>
diff --git a/src/bin/pg_rewind/Makefile b/src/bin/pg_rewind/Makefile
index 859d3abc41..98fb381c62 100644
--- a/src/bin/pg_rewind/Makefile
+++ b/src/bin/pg_rewind/Makefile
@@ -16,7 +16,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I$(libpq_srcdir) -DFRONTEND $(CPPFLAGS)
-LDFLAGS_INTERNAL += $(libpq_pgport)
+LDFLAGS_INTERNAL += $(libpq_pgport) -L$(top_builddir)/src/fe_utils -lpgfeutils
OBJS = pg_rewind.o parsexlog.o xlogreader.o datapagemap.o timeline.o \
fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o \
diff --git a/src/bin/pg_rewind/libpq_fetch.c b/src/bin/pg_rewind/libpq_fetch.c
index 002776f696..f4ebf7d842 100644
--- a/src/bin/pg_rewind/libpq_fetch.c
+++ b/src/bin/pg_rewind/libpq_fetch.c
@@ -20,12 +20,11 @@
#include "file_ops.h"
#include "filemap.h"
-#include "libpq-fe.h"
#include "catalog/pg_type_d.h"
#include "fe_utils/connect.h"
#include "port/pg_bswap.h"
-static PGconn *conn = NULL;
+PGconn *conn = NULL;
/*
* Files are fetched max CHUNKSIZE bytes at a time.
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index 15e3eab550..6f1dadbceb 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -27,6 +27,7 @@
#include "common/file_perm.h"
#include "common/file_utils.h"
#include "common/restricted_token.h"
+#include "fe_utils/recovery_gen.h"
#include "getopt_long.h"
#include "storage/bufpage.h"
@@ -65,6 +66,12 @@ int targetNentries;
uint64 fetch_size;
uint64 fetch_done;
+static void
+disconnect_atexit(void)
+{
+ if (conn != NULL)
+ PQfinish(conn);
+}
static void
usage(const char *progname)
@@ -75,6 +82,7 @@ usage(const char *progname)
printf(_(" -D, --target-pgdata=DIRECTORY existing data directory to modify\n"));
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
+ printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -92,6 +100,7 @@ main(int argc, char **argv)
static struct option long_options[] = {
{"help", no_argument, NULL, '?'},
{"target-pgdata", required_argument, NULL, 'D'},
+ {"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
{"version", no_argument, NULL, 'V'},
@@ -114,6 +123,7 @@ main(int argc, char **argv)
XLogRecPtr endrec;
TimeLineID endtli;
ControlFileData ControlFile_new;
+ bool writerecoveryconf = false;
pg_logging_init(argv[0]);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_rewind"));
@@ -134,7 +144,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNP", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -154,6 +164,10 @@ main(int argc, char **argv)
do_sync = false;
break;
+ case 'R':
+ writerecoveryconf = true;
+ break;
+
case 3:
debug = true;
pg_logging_set_level(PG_LOG_DEBUG);
@@ -229,6 +243,8 @@ main(int argc, char **argv)
umask(pg_mode_mask);
+ atexit(disconnect_atexit);
+
/* Connect to remote server */
if (connstr_source)
libpqConnect(connstr_source);
@@ -297,6 +313,9 @@ main(int argc, char **argv)
if (!rewind_needed)
{
pg_log_info("no rewind required");
+ if (writerecoveryconf && connstr_source)
+ WriteRecoveryConfig(conn, datadir_target,
+ GenerateRecoveryConfig(conn, NULL));
exit(0);
}
@@ -394,6 +413,10 @@ main(int argc, char **argv)
pg_log_info("syncing target data directory");
syncTargetDirectory();
+ if (writerecoveryconf && connstr_source)
+ WriteRecoveryConfig(conn, datadir_target,
+ GenerateRecoveryConfig(conn, NULL));
+
pg_log_info("Done!");
return 0;
diff --git a/src/bin/pg_rewind/pg_rewind.h b/src/bin/pg_rewind/pg_rewind.h
index 1125c7e60f..46aa6df4ba 100644
--- a/src/bin/pg_rewind/pg_rewind.h
+++ b/src/bin/pg_rewind/pg_rewind.h
@@ -16,9 +16,10 @@
#include "access/timeline.h"
#include "storage/block.h"
#include "storage/relfilenode.h"
-
#include "common/logging.h"
+#include "libpq-fe.h"
+
/* Configuration options */
extern char *datadir_target;
extern char *datadir_source;
@@ -26,6 +27,7 @@ extern char *connstr_source;
extern bool showprogress;
extern bool dry_run;
extern int WalSegSz;
+extern PGconn *conn;
/* Target history */
extern TimeLineHistoryEntry *targetHistory;
--
2.17.2
v9-0002-Ensure-target-clean-shutdown-in-pg_rewind.patch (application/octet-stream, 6.0K) download
From 30082eab047e677a8b17ba255f512e04ca1f21ef Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Tue, 19 Mar 2019 12:41:30 +0800
Subject: [PATCH v9 2/2] Ensure target clean shutdown in pg_rewind
---
doc/src/sgml/ref/pg_rewind.sgml | 16 ++++++++
src/bin/pg_rewind/pg_rewind.c | 72 ++++++++++++++++++++++++++++++++-
2 files changed, 87 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index b8b43e8220..aa269b3892 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -176,6 +176,22 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-s</option></term>
+ <term><option>--no-ensure-shutdown</option></term>
+ <listitem>
+ <para>
+ By default, <application>pg_rewind</application> makes sure target is
+ cleanly shutdown before doing recovery by launching a single mode
+ postgres instance if needed to ensure the target has completed crash
+ recovery and pg_control is in clean shutdown state. Clean shutdown
+ state is a prerequisite of <application>pg_rewind</application>.
+ With this option, <application>pg_rewind</application> skips this and
+ then users need to do this themselves.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-n</option></term>
<term><option>--dry-run</option></term>
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index 6f1dadbceb..d41bb12ade 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -41,6 +41,7 @@ static void digestControlFile(ControlFileData *ControlFile, char *source,
static void syncTargetDirectory(void);
static void sanityChecks(void);
static void findCommonAncestorTimeline(XLogRecPtr *recptr, int *tliIndex);
+static void ensureCleanShutdown(const char *argv0);
static ControlFileData ControlFile_target;
static ControlFileData ControlFile_source;
@@ -83,6 +84,7 @@ usage(const char *progname)
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
+ printf(_(" -s, --no-ensure-shutdown do not automatically fix unclean shutdown\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -103,6 +105,7 @@ main(int argc, char **argv)
{"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
+ {"no-ensure-shutdown", no_argument, NULL, 's'},
{"version", no_argument, NULL, 'V'},
{"dry-run", no_argument, NULL, 'n'},
{"no-sync", no_argument, NULL, 'N'},
@@ -119,6 +122,7 @@ main(int argc, char **argv)
XLogRecPtr chkptredo;
size_t size;
char *buffer;
+ bool no_ensure_shutdown = false;
bool rewind_needed;
XLogRecPtr endrec;
TimeLineID endtli;
@@ -144,7 +148,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPRs", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -152,6 +156,10 @@ main(int argc, char **argv)
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
+ case 's':
+ no_ensure_shutdown = true;
+ break;
+
case 'P':
showprogress = true;
break;
@@ -257,6 +265,24 @@ main(int argc, char **argv)
digestControlFile(&ControlFile_target, buffer, size);
pg_free(buffer);
+ /*
+ * If the target instance was not cleanly shut down, run a single-user
+ * postgres session really quickly and reload the control file to get the
+ * new state. Note if no_ensure_shutdown is specified, pg_rewind won't
+ * do that automatically. That means users need to do themselves in
+ * advance, else pg_rewind will soon quit, see sanityChecks().
+ */
+ if (!no_ensure_shutdown &&
+ ControlFile_target.state != DB_SHUTDOWNED &&
+ ControlFile_target.state != DB_SHUTDOWNED_IN_RECOVERY)
+ {
+ ensureCleanShutdown(argv[0]);
+
+ buffer = slurpFile(datadir_target, "global/pg_control", &size);
+ digestControlFile(&ControlFile_target, buffer, size);
+ pg_free(buffer);
+ }
+
buffer = fetchFile("global/pg_control", &size);
digestControlFile(&ControlFile_source, buffer, size);
pg_free(buffer);
@@ -771,3 +797,47 @@ syncTargetDirectory(void)
fsync_pgdata(datadir_target, PG_VERSION_NUM);
}
+
+/*
+ * Ensure clean shutdown of target instance by launching single-user mode
+ * postgres to do crash recovery.
+ */
+static void
+ensureCleanShutdown(const char *argv0)
+{
+ int ret;
+#define MAXCMDLEN (2 * MAXPGPATH)
+ char exec_path[MAXPGPATH];
+ char cmd[MAXCMDLEN];
+
+ /* locate postgres binary */
+ if ((ret = find_other_exec(argv0, "postgres",
+ PG_BACKEND_VERSIONSTR,
+ exec_path)) < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(argv0, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+
+ if (ret == -1)
+ pg_fatal("The program \"postgres\" is needed by %s but was \n"
+ "not found in the same directory as \"%s\".\n"
+ "Check your installation.\n", progname, full_path);
+ else
+ pg_fatal("The program \"postgres\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.\n", full_path, progname);
+ }
+
+ /* only skip processing after ensuring presence of postgres */
+ if (dry_run)
+ return;
+
+ /* finally run postgres single-user mode */
+ snprintf(cmd, MAXCMDLEN, "\"%s\" --single -D \"%s\" template1 < %s",
+ exec_path, datadir_target, DEVNULL);
+
+ if (system(cmd) != 0)
+ pg_fatal("postgres single-user mode of target instance failed for command: %s\n", cmd);
+}
--
2.17.2
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-09-27 12:18 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-27 14:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-30 07:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
1 sibling, 2 replies; 46+ messages in thread
From: Alexey Kondratov @ 2019-09-27 12:18 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On 27.09.2019 6:27, Paul Guo wrote:
>
>
> Secondarily, I see no reason to test connstr_source rather than just
> "conn" in the other patch; doing it the other way is more natural,
> since
> it's that thing that's tested as an argument.
>
> pg_rewind.c: Please put the new #include line keeping the alphabetical
> order.
>
>
> Agreed to the above suggestions. I attached the v9.
>
I went through the remaining two patches and they seem to be very clear
and concise. However, there are two points I could complain about:
1) Maybe I've missed it somewhere in the thread above, but currently
pg_rewind allows to run itself with -R and --source-pgdata. In that case
-R option is just swallowed and neither standby.signal, nor
postgresql.auto.conf is written, which is reasonable though. Should it
be stated somehow in the docs that -R option always has to go altogether
with --source-server? Or should pg_rewind notify user that options are
incompatible and no recovery configuration will be written?
2) Are you going to leave -R option completely without tap-tests?
Attached is a small patch, which tests -R option along with the existing
'remote' case. If needed it may be split into two separate cases. First,
it tests that pg_rewind is able to succeed with minimal permissions
according to the Michael's patch d9f543e [1]. Next, it checks presence
of standby.signal and adds REPLICATION permission to rewind_user to test
that new standby is able to start with generated recovery configuration.
[1]
https://github.com/postgres/postgres/commit/d9f543e9e9be15f92abdeaf870e57ef289020191
Regards
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
Attachments:
v9-0003-Test-new-standby-start-with-generated-config-duri.patch (text/x-patch, 2.8K) download
From 8c607794f259cd4dec0fa6172b69d62e6468bee3 Mon Sep 17 00:00:00 2001
From: Alexey Kondratov <kondratov.aleksey@gmail.com>
Date: Fri, 27 Sep 2019 14:30:57 +0300
Subject: [PATCH v9 3/3] Test new standby start with generated config during
pg_rewind remote
---
src/bin/pg_rewind/t/001_basic.pl | 2 +-
src/bin/pg_rewind/t/002_databases.pl | 2 +-
src/bin/pg_rewind/t/003_extrafiles.pl | 2 +-
src/bin/pg_rewind/t/004_pg_xlog_symlink.pl | 2 +-
src/bin/pg_rewind/t/RewindTest.pm | 11 ++++++++++-
5 files changed, 14 insertions(+), 5 deletions(-)
diff --git a/src/bin/pg_rewind/t/001_basic.pl b/src/bin/pg_rewind/t/001_basic.pl
index 115192170e..c3293e93df 100644
--- a/src/bin/pg_rewind/t/001_basic.pl
+++ b/src/bin/pg_rewind/t/001_basic.pl
@@ -1,7 +1,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 10;
+use Test::More tests => 11;
use FindBin;
use lib $FindBin::RealBin;
diff --git a/src/bin/pg_rewind/t/002_databases.pl b/src/bin/pg_rewind/t/002_databases.pl
index f1eb4fe1d2..1db534c0dc 100644
--- a/src/bin/pg_rewind/t/002_databases.pl
+++ b/src/bin/pg_rewind/t/002_databases.pl
@@ -1,7 +1,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 6;
+use Test::More tests => 7;
use FindBin;
use lib $FindBin::RealBin;
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index c4040bd562..f4710440fc 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -3,7 +3,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 4;
+use Test::More tests => 5;
use File::Find;
diff --git a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
index ed1ddb6b60..639eeb9c91 100644
--- a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
+++ b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
@@ -14,7 +14,7 @@ if ($windows_os)
}
else
{
- plan tests => 4;
+ plan tests => 5;
}
use FindBin;
diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm
index 68b6004e94..fcc48cb1d9 100644
--- a/src/bin/pg_rewind/t/RewindTest.pm
+++ b/src/bin/pg_rewind/t/RewindTest.pm
@@ -266,9 +266,18 @@ sub run_pg_rewind
[
'pg_rewind', "--debug",
"--source-server", $standby_connstr,
- "--target-pgdata=$master_pgdata", "--no-sync"
+ "--target-pgdata=$master_pgdata", "-R", "--no-sync"
],
'pg_rewind remote');
+
+ # Check that standby.signal has been created.
+ ok(-e "$master_pgdata/standby.signal");
+
+ # Now, when pg_rewind apparently succeeded with minimal permissions,
+ # add REPLICATION privilege. So we could test that new standby
+ # is able to connect to the new master with generated config.
+ $node_standby->psql(
+ 'postgres', "ALTER ROLE rewind_user WITH REPLICATION;");
}
else
{
--
2.17.1
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-27 12:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
@ 2019-09-27 14:28 ` Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 0 replies; 46+ messages in thread
From: Alvaro Herrera @ 2019-09-27 14:28 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; +Cc: Paul Guo <pguo@pivotal.io>; Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On 2019-Sep-27, Alexey Kondratov wrote:
> 1) Maybe I've missed it somewhere in the thread above, but currently
> pg_rewind allows to run itself with -R and --source-pgdata. In that case -R
> option is just swallowed and neither standby.signal, nor
> postgresql.auto.conf is written, which is reasonable though. Should it be
> stated somehow in the docs that -R option always has to go altogether with
> --source-server? Or should pg_rewind notify user that options are
> incompatible and no recovery configuration will be written?
Hmm I think it should throw an error, yeah. Ignoring options is not
good.
> + # Now, when pg_rewind apparently succeeded with minimal permissions,
> + # add REPLICATION privilege. So we could test that new standby
> + # is able to connect to the new master with generated config.
> + $node_standby->psql(
> + 'postgres', "ALTER ROLE rewind_user WITH REPLICATION;");
I think this better use safe_psql.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-27 12:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
@ 2019-09-30 07:07 ` Paul Guo <pguo@pivotal.io>
2019-09-30 08:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-30 17:13 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 2 replies; 46+ messages in thread
From: Paul Guo @ 2019-09-30 07:07 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
>
>
> I went through the remaining two patches and they seem to be very clear
> and concise. However, there are two points I could complain about:
>
> 1) Maybe I've missed it somewhere in the thread above, but currently
> pg_rewind allows to run itself with -R and --source-pgdata. In that case
> -R option is just swallowed and neither standby.signal, nor
> postgresql.auto.conf is written, which is reasonable though. Should it
> be stated somehow in the docs that -R option always has to go altogether
> with --source-server? Or should pg_rewind notify user that options are
> incompatible and no recovery configuration will be written?
>
I modified code & doc to address this. In code, pg_rewind will error out
for the local case.
> 2) Are you going to leave -R option completely without tap-tests?
> Attached is a small patch, which tests -R option along with the existing
> 'remote' case. If needed it may be split into two separate cases. First,
> it tests that pg_rewind is able to succeed with minimal permissions
> according to the Michael's patch d9f543e [1]. Next, it checks presence
> of standby.signal and adds REPLICATION permission to rewind_user to test
> that new standby is able to start with generated recovery configuration.
>
> [1]
>
> https://github.com/postgres/postgres/commit/d9f543e9e9be15f92abdeaf870e57ef289020191
>
>
It seems that we could further disabling recovery info setting code for the
'remote' test case?
- my $port_standby = $node_standby->port;
- $node_master->append_conf(
- 'postgresql.conf', qq(
-primary_conninfo='port=$port_standby'
-));
+ if ($test_mode ne "remote")
+ {
+ my $port_standby = $node_standby->port;
+ $node_master->append_conf(
+ 'postgresql.conf',
+ qq(primary_conninfo='port=$port_standby'));
- $node_master->set_standby_mode();
+ $node_master->set_standby_mode();
+ }
Thanks.
Attachments:
v10-0001-Add-option-to-write-recovery-configuration-infor.patch (application/octet-stream, 6.4K) download
From 18000dc65214587fd05989de99b576c353ddba6f Mon Sep 17 00:00:00 2001
From: Paul Guo <paulguo@gmail.com>
Date: Thu, 18 Apr 2019 19:37:36 +0800
Subject: [PATCH v10] Add option to write recovery configuration information in
pg_rewind.
---
doc/src/sgml/ref/pg_rewind.sgml | 12 ++++++++++++
src/bin/pg_rewind/Makefile | 2 +-
src/bin/pg_rewind/libpq_fetch.c | 3 +--
src/bin/pg_rewind/pg_rewind.c | 33 ++++++++++++++++++++++++++++++++-
src/bin/pg_rewind/pg_rewind.h | 4 +++-
5 files changed, 49 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index a06e5ac5e1..a5fdbcd4d5 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -180,6 +180,18 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-R</option></term>
+ <term><option>--write-recovery-conf</option></term>
+ <listitem>
+ <para>
+ Create <filename>standby.signal</filename> and append connection settings
+ to <filename>postgresql.auto.conf</filename> in the output directory.
+ It is only used by the case with source server.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-n</option></term>
<term><option>--dry-run</option></term>
diff --git a/src/bin/pg_rewind/Makefile b/src/bin/pg_rewind/Makefile
index 859d3abc41..98fb381c62 100644
--- a/src/bin/pg_rewind/Makefile
+++ b/src/bin/pg_rewind/Makefile
@@ -16,7 +16,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I$(libpq_srcdir) -DFRONTEND $(CPPFLAGS)
-LDFLAGS_INTERNAL += $(libpq_pgport)
+LDFLAGS_INTERNAL += $(libpq_pgport) -L$(top_builddir)/src/fe_utils -lpgfeutils
OBJS = pg_rewind.o parsexlog.o xlogreader.o datapagemap.o timeline.o \
fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o \
diff --git a/src/bin/pg_rewind/libpq_fetch.c b/src/bin/pg_rewind/libpq_fetch.c
index 002776f696..f4ebf7d842 100644
--- a/src/bin/pg_rewind/libpq_fetch.c
+++ b/src/bin/pg_rewind/libpq_fetch.c
@@ -20,12 +20,11 @@
#include "file_ops.h"
#include "filemap.h"
-#include "libpq-fe.h"
#include "catalog/pg_type_d.h"
#include "fe_utils/connect.h"
#include "port/pg_bswap.h"
-static PGconn *conn = NULL;
+PGconn *conn = NULL;
/*
* Files are fetched max CHUNKSIZE bytes at a time.
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index 8cb0d726cf..63a2059021 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -27,6 +27,7 @@
#include "common/file_perm.h"
#include "common/file_utils.h"
#include "common/restricted_token.h"
+#include "fe_utils/recovery_gen.h"
#include "getopt_long.h"
#include "storage/bufpage.h"
@@ -66,6 +67,12 @@ int targetNentries;
uint64 fetch_size;
uint64 fetch_done;
+static void
+disconnect_atexit(void)
+{
+ if (conn != NULL)
+ PQfinish(conn);
+}
static void
usage(const char *progname)
@@ -76,6 +83,7 @@ usage(const char *progname)
printf(_(" -D, --target-pgdata=DIRECTORY existing data directory to modify\n"));
printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n"));
printf(_(" --source-server=CONNSTR source server to synchronize with\n"));
+ printf(_(" -R, --write-recovery-conf write configuration for replication\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
printf(_(" -N, --no-sync do not wait for changes to be written\n"
" safely to disk\n"));
@@ -94,6 +102,7 @@ main(int argc, char **argv)
static struct option long_options[] = {
{"help", no_argument, NULL, '?'},
{"target-pgdata", required_argument, NULL, 'D'},
+ {"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
{"no-ensure-shutdown", no_argument, NULL, 44},
@@ -118,6 +127,7 @@ main(int argc, char **argv)
XLogRecPtr endrec;
TimeLineID endtli;
ControlFileData ControlFile_new;
+ bool writerecoveryconf = false;
pg_logging_init(argv[0]);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_rewind"));
@@ -138,7 +148,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:nNP", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "D:nNPR", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -158,6 +168,10 @@ main(int argc, char **argv)
do_sync = false;
break;
+ case 'R':
+ writerecoveryconf = true;
+ break;
+
case 3:
debug = true;
pg_logging_set_level(PG_LOG_DEBUG);
@@ -200,6 +214,14 @@ main(int argc, char **argv)
exit(1);
}
+ if (writerecoveryconf && connstr_source == NULL)
+ {
+ pg_log_error("no source server information is specified "
+ "(--source--server) for -R or --write-recovery-conf");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ exit(1);
+ }
+
if (optind < argc)
{
pg_log_error("too many command-line arguments (first is \"%s\")",
@@ -236,6 +258,8 @@ main(int argc, char **argv)
umask(pg_mode_mask);
+ atexit(disconnect_atexit);
+
/* Connect to remote server */
if (connstr_source)
libpqConnect(connstr_source);
@@ -322,6 +346,9 @@ main(int argc, char **argv)
if (!rewind_needed)
{
pg_log_info("no rewind required");
+ if (writerecoveryconf)
+ WriteRecoveryConfig(conn, datadir_target,
+ GenerateRecoveryConfig(conn, NULL));
exit(0);
}
@@ -419,6 +446,10 @@ main(int argc, char **argv)
pg_log_info("syncing target data directory");
syncTargetDirectory();
+ if (writerecoveryconf)
+ WriteRecoveryConfig(conn, datadir_target,
+ GenerateRecoveryConfig(conn, NULL));
+
pg_log_info("Done!");
return 0;
diff --git a/src/bin/pg_rewind/pg_rewind.h b/src/bin/pg_rewind/pg_rewind.h
index 1125c7e60f..46aa6df4ba 100644
--- a/src/bin/pg_rewind/pg_rewind.h
+++ b/src/bin/pg_rewind/pg_rewind.h
@@ -16,9 +16,10 @@
#include "access/timeline.h"
#include "storage/block.h"
#include "storage/relfilenode.h"
-
#include "common/logging.h"
+#include "libpq-fe.h"
+
/* Configuration options */
extern char *datadir_target;
extern char *datadir_source;
@@ -26,6 +27,7 @@ extern char *connstr_source;
extern bool showprogress;
extern bool dry_run;
extern int WalSegSz;
+extern PGconn *conn;
/* Target history */
extern TimeLineHistoryEntry *targetHistory;
--
2.17.2
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-27 12:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-30 07:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-09-30 08:51 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
1 sibling, 0 replies; 46+ messages in thread
From: Alexey Kondratov @ 2019-09-30 08:51 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On 27.09.2019 17:28, Alvaro Herrera wrote:
>
>> + # Now, when pg_rewind apparently succeeded with minimal permissions,
>> + # add REPLICATION privilege. So we could test that new standby
>> + # is able to connect to the new master with generated config.
>> + $node_standby->psql(
>> + 'postgres', "ALTER ROLE rewind_user WITH REPLICATION;");
> I think this better use safe_psql.
>
Yes, indeed.
On 30.09.2019 10:07, Paul Guo wrote:
>
> 2) Are you going to leave -R option completely without tap-tests?
> Attached is a small patch, which tests -R option along with the
> existing
> 'remote' case. If needed it may be split into two separate cases.
> First,
> it tests that pg_rewind is able to succeed with minimal permissions
> according to the Michael's patch d9f543e [1]. Next, it checks
> presence
> of standby.signal and adds REPLICATION permission to rewind_user
> to test
> that new standby is able to start with generated recovery
> configuration.
>
> [1]
> https://github.com/postgres/postgres/commit/d9f543e9e9be15f92abdeaf870e57ef289020191
>
> It seems that we could further disabling recovery info setting code
> for the 'remote' test case?
>
> - my $port_standby = $node_standby->port;
> - $node_master->append_conf(
> - 'postgresql.conf', qq(
> -primary_conninfo='port=$port_standby'
> -));
> + if ($test_mode ne "remote")
> + {
> + my $port_standby = $node_standby->port;
> + $node_master->append_conf(
> + 'postgresql.conf',
> + qq(primary_conninfo='port=$port_standby'));
>
> - $node_master->set_standby_mode();
> + $node_master->set_standby_mode();
> + }
>
>
Yeah, it makes sense. It is excessive for remote if we add '-R' there.
I've updated and attached my test adding patch.
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
Attachments:
v10-0003-Test-new-standby-start-with-generated-config-dur.patch (text/x-patch, 3.8K) download
From b38bc7d71f7e7d68d66d3bf9af4e6371445aeab2 Mon Sep 17 00:00:00 2001
From: Alexey Kondratov <kondratov.aleksey@gmail.com>
Date: Fri, 27 Sep 2019 14:30:57 +0300
Subject: [PATCH v10 3/3] Test new standby start with generated config during
pg_rewind remote
---
src/bin/pg_rewind/t/001_basic.pl | 2 +-
src/bin/pg_rewind/t/002_databases.pl | 2 +-
src/bin/pg_rewind/t/003_extrafiles.pl | 2 +-
src/bin/pg_rewind/t/004_pg_xlog_symlink.pl | 2 +-
src/bin/pg_rewind/t/RewindTest.pm | 27 +++++++++++++++-------
5 files changed, 23 insertions(+), 12 deletions(-)
diff --git a/src/bin/pg_rewind/t/001_basic.pl b/src/bin/pg_rewind/t/001_basic.pl
index 115192170e..c3293e93df 100644
--- a/src/bin/pg_rewind/t/001_basic.pl
+++ b/src/bin/pg_rewind/t/001_basic.pl
@@ -1,7 +1,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 10;
+use Test::More tests => 11;
use FindBin;
use lib $FindBin::RealBin;
diff --git a/src/bin/pg_rewind/t/002_databases.pl b/src/bin/pg_rewind/t/002_databases.pl
index f1eb4fe1d2..1db534c0dc 100644
--- a/src/bin/pg_rewind/t/002_databases.pl
+++ b/src/bin/pg_rewind/t/002_databases.pl
@@ -1,7 +1,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 6;
+use Test::More tests => 7;
use FindBin;
use lib $FindBin::RealBin;
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index c4040bd562..f4710440fc 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -3,7 +3,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 4;
+use Test::More tests => 5;
use File::Find;
diff --git a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
index ed1ddb6b60..639eeb9c91 100644
--- a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
+++ b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
@@ -14,7 +14,7 @@ if ($windows_os)
}
else
{
- plan tests => 4;
+ plan tests => 5;
}
use FindBin;
diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm
index 68b6004e94..2b45c2789c 100644
--- a/src/bin/pg_rewind/t/RewindTest.pm
+++ b/src/bin/pg_rewind/t/RewindTest.pm
@@ -149,7 +149,7 @@ sub start_master
# Create custom role which is used to run pg_rewind, and adjust its
# permissions to the minimum necessary.
- $node_master->psql(
+ $node_master->safe_psql(
'postgres', "
CREATE ROLE rewind_user LOGIN;
GRANT EXECUTE ON function pg_catalog.pg_ls_dir(text, boolean, boolean)
@@ -266,9 +266,18 @@ sub run_pg_rewind
[
'pg_rewind', "--debug",
"--source-server", $standby_connstr,
- "--target-pgdata=$master_pgdata", "--no-sync"
+ "--target-pgdata=$master_pgdata", "-R", "--no-sync"
],
'pg_rewind remote');
+
+ # Check that standby.signal has been created.
+ ok(-e "$master_pgdata/standby.signal");
+
+ # Now, when pg_rewind apparently succeeded with minimal permissions,
+ # add REPLICATION privilege. So we could test that new standby
+ # is able to connect to the new master with generated config.
+ $node_standby->safe_psql(
+ 'postgres', "ALTER ROLE rewind_user WITH REPLICATION;");
}
else
{
@@ -289,13 +298,15 @@ sub run_pg_rewind
"unable to set permissions for $master_pgdata/postgresql.conf");
# Plug-in rewound node to the now-promoted standby node
- my $port_standby = $node_standby->port;
- $node_master->append_conf(
- 'postgresql.conf', qq(
-primary_conninfo='port=$port_standby'
-));
+ if ($test_mode ne "remote")
+ {
+ my $port_standby = $node_standby->port;
+ $node_master->append_conf(
+ 'postgresql.conf', qq(
+primary_conninfo='port=$port_standby'));
- $node_master->set_standby_mode();
+ $node_master->set_standby_mode();
+ }
# Restart the master to check that rewind went correctly
$node_master->start;
--
2.17.1
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-27 12:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-30 07:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-09-30 17:13 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-10-02 17:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
1 sibling, 1 reply; 46+ messages in thread
From: Alvaro Herrera @ 2019-09-30 17:13 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: Alexey Kondratov <a.kondratov@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
OK, I pushed this patch as well as Alexey's test patch. It all works
for me, and the coverage report shows that we're doing the new thing ...
though only in the case that rewind *is* required. There is no test to
verify the case where rewind is *not* required. I guess it'd also be
good to test the case when we throw the new error, if only for
completeness ...
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-27 12:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-30 07:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-30 17:13 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2019-10-02 17:28 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-03 03:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 46+ messages in thread
From: Alexey Kondratov @ 2019-10-02 17:28 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; Paul Guo <pguo@pivotal.io>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
Hi Alvaro,
On 30.09.2019 20:13, Alvaro Herrera wrote:
> OK, I pushed this patch as well as Alexey's test patch. It all works
> for me, and the coverage report shows that we're doing the new thing ...
> though only in the case that rewind *is* required. There is no test to
> verify the case where rewind is *not* required. I guess it'd also be
> good to test the case when we throw the new error, if only for
> completeness ...
I've directly followed your guess and tried to elaborate pg_rewind test
cases and... It seems I've caught a few bugs:
1) --dry-run actually wasn't completely 'dry'. It did update target
controlfile, which could cause repetitive pg_rewind calls to fail after
dry-run ones.
2) --no-ensure-shutdown flag was broken, it simply didn't turn off this
new feature.
3) --write-recovery-conf didn't obey the --dry-run flag.
Thus, it was definitely a good idea to add new tests. Two patches are
attached:
1) First one fixes all the issues above;
2) Second one slightly increases pg_rewind overall code coverage from
74% to 78.6%.
Should I put this fix on the next commitfest?
Regards
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
P.S. My apologies that I've missed two of these bugs during review.
Attachments: download all (2 files, .tar.gz)
v1-0001-Fix-functionality-of-pg_rewind-dry-run-and-no-ens.patch (text/x-patch, 1.6K) download
From 7286e31ab0ebf50bb4ab460dd81b82f1c5989272 Mon Sep 17 00:00:00 2001
From: Alexey Kondratov <kondratov.aleksey@gmail.com>
Date: Wed, 2 Oct 2019 19:24:46 +0300
Subject: [PATCH v1 1/2] Fix functionality of pg_rewind --dry-run and
--no-ensure-shutdown options
Branch: pg-rewind-fixes
---
src/bin/pg_rewind/pg_rewind.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index a7fd9e0cab..1a7fb5242b 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -101,7 +101,7 @@ main(int argc, char **argv)
{"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
- {"no-ensure-shutdown", no_argument, NULL, 44},
+ {"no-ensure-shutdown", no_argument, NULL, 4},
{"version", no_argument, NULL, 'V'},
{"dry-run", no_argument, NULL, 'n'},
{"no-sync", no_argument, NULL, 'N'},
@@ -435,13 +435,15 @@ main(int argc, char **argv)
ControlFile_new.minRecoveryPoint = endrec;
ControlFile_new.minRecoveryPointTLI = endtli;
ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY;
- update_controlfile(datadir_target, &ControlFile_new, do_sync);
+
+ if (!dry_run)
+ update_controlfile(datadir_target, &ControlFile_new, do_sync);
if (showprogress)
pg_log_info("syncing target data directory");
syncTargetDirectory();
- if (writerecoveryconf)
+ if (!dry_run && writerecoveryconf)
WriteRecoveryConfig(conn, datadir_target,
GenerateRecoveryConfig(conn, NULL));
base-commit: df86e52cace2c4134db51de6665682fb985f3195
--
2.17.1
v1-0002-Increase-pg_rewind-code-coverage.patch (text/x-patch, 4.8K) download
From 28fdd2fa58af718d8a894cb3c3d8f9b2cdf6759e Mon Sep 17 00:00:00 2001
From: Alexey Kondratov <kondratov.aleksey@gmail.com>
Date: Wed, 2 Oct 2019 19:25:27 +0300
Subject: [PATCH v1 2/2] Increase pg_rewind code coverage
Branch: pg-rewind-fixes
---
src/bin/pg_rewind/t/001_basic.pl | 2 +-
src/bin/pg_rewind/t/002_databases.pl | 2 +-
src/bin/pg_rewind/t/003_extrafiles.pl | 2 +-
src/bin/pg_rewind/t/004_pg_xlog_symlink.pl | 2 +-
src/bin/pg_rewind/t/005_same_timeline.pl | 27 ++++++++++++++----
src/bin/pg_rewind/t/RewindTest.pm | 33 ++++++++++++++++++++--
6 files changed, 55 insertions(+), 13 deletions(-)
diff --git a/src/bin/pg_rewind/t/001_basic.pl b/src/bin/pg_rewind/t/001_basic.pl
index c3293e93df..a1659460ec 100644
--- a/src/bin/pg_rewind/t/001_basic.pl
+++ b/src/bin/pg_rewind/t/001_basic.pl
@@ -1,7 +1,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 11;
+use Test::More tests => 14;
use FindBin;
use lib $FindBin::RealBin;
diff --git a/src/bin/pg_rewind/t/002_databases.pl b/src/bin/pg_rewind/t/002_databases.pl
index 1db534c0dc..921c4434f5 100644
--- a/src/bin/pg_rewind/t/002_databases.pl
+++ b/src/bin/pg_rewind/t/002_databases.pl
@@ -1,7 +1,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 7;
+use Test::More tests => 10;
use FindBin;
use lib $FindBin::RealBin;
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index f4710440fc..bce5b47148 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -3,7 +3,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 5;
+use Test::More tests => 8;
use File::Find;
diff --git a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
index 639eeb9c91..a501be8f78 100644
--- a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
+++ b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
@@ -14,7 +14,7 @@ if ($windows_os)
}
else
{
- plan tests => 5;
+ plan tests => 8;
}
use FindBin;
diff --git a/src/bin/pg_rewind/t/005_same_timeline.pl b/src/bin/pg_rewind/t/005_same_timeline.pl
index 40dbc44caa..142e675a23 100644
--- a/src/bin/pg_rewind/t/005_same_timeline.pl
+++ b/src/bin/pg_rewind/t/005_same_timeline.pl
@@ -1,7 +1,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 1;
+use Test::More tests => 6;
use FindBin;
use lib $FindBin::RealBin;
@@ -11,9 +11,24 @@ use RewindTest;
# Test that running pg_rewind if the two clusters are on the same
# timeline runs successfully.
-RewindTest::setup_cluster();
-RewindTest::start_master();
-RewindTest::create_standby();
-RewindTest::run_pg_rewind('local');
-RewindTest::clean_rewind_test();
+sub run_test
+{
+ my $test_mode = shift;
+
+ RewindTest::setup_cluster($test_mode, ['-g']);
+ RewindTest::start_master();
+ RewindTest::create_standby($test_mode);
+
+ RewindTest::promote_standby();
+
+ RewindTest::run_pg_rewind($test_mode);
+
+ RewindTest::clean_rewind_test();
+ return;
+}
+
+# Run the test in both modes.
+run_test('local');
+run_test('remote');
+
exit(0);
diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm
index c540722420..497807718c 100644
--- a/src/bin/pg_rewind/t/RewindTest.pm
+++ b/src/bin/pg_rewind/t/RewindTest.pm
@@ -245,9 +245,20 @@ sub run_pg_rewind
if ($test_mode eq "local")
{
- # Do rewind using a local pgdata as source
- # Stop the master and be ready to perform the rewind
+ # Stop the new master and be ready to perform the rewind
$node_standby->stop;
+
+ # Check that incompatible options error out.
+ command_fails(
+ [
+ 'pg_rewind', "--debug",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata", "-R",
+ "--no-ensure-shutdown"
+ ],
+ 'pg_rewind local with -R');
+
+ # Do rewind using a local pgdata as source
command_ok(
[
'pg_rewind',
@@ -261,6 +272,21 @@ sub run_pg_rewind
elsif ($test_mode eq "remote")
{
+ # Do rewind in dry-run mode using a remote connection as source
+ command_ok(
+ [
+ 'pg_rewind',
+ "--source-server", $standby_connstr,
+ "--target-pgdata=$master_pgdata", "-R",
+ "--debug", "--no-ensure-shutdown", "-P",
+ "--no-sync", "--dry-run"
+ ],
+ 'pg_rewind remote with --dry-run');
+
+ # Check that standby.signal hasn't been created.
+ ok(! -e "$master_pgdata/standby.signal",
+ 'standby.signal is absent');
+
# Do rewind using a remote connection as source
command_ok(
[
@@ -272,7 +298,8 @@ sub run_pg_rewind
'pg_rewind remote');
# Check that standby.signal has been created.
- ok(-e "$master_pgdata/standby.signal");
+ ok(-e "$master_pgdata/standby.signal",
+ 'standby.signal exists');
# Now, when pg_rewind apparently succeeded with minimal permissions,
# add REPLICATION privilege. So we could test that new standby
--
2.17.1
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-27 12:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-30 07:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-30 17:13 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-10-02 17:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
@ 2019-10-03 03:07 ` Michael Paquier <michael@paquier.xyz>
2019-10-03 09:43 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
0 siblings, 1 reply; 46+ messages in thread
From: Michael Paquier @ 2019-10-03 03:07 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Paul Guo <pguo@pivotal.io>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On Wed, Oct 02, 2019 at 08:28:09PM +0300, Alexey Kondratov wrote:
> I've directly followed your guess and tried to elaborate pg_rewind test
> cases and... It seems I've caught a few bugs:
>
> 1) --dry-run actually wasn't completely 'dry'. It did update target
> controlfile, which could cause repetitive pg_rewind calls to fail after
> dry-run ones.
I have just paid attention to this thread, but this is a bug which
goes down to 12 actually so let's treat it independently of the rest.
The control file was not written thanks to the safeguards in
write_target_range() in past versions, but the recent refactoring
around control file handling broke that promise. Another thing which
is not completely exact is the progress reporting which should be
reported even if the dry-run mode runs. That's less critical, but
let's make things consistent.
Patch 0001 also forgot that recovery.conf should not be written either
when no rewind is needed.
I have reworked your first patch as per the attached. What do you
think about it? The part with the control file needs to go down to
v12, and I would likely split that into two commits on HEAD: one for
the control file and a second for the recovery.conf portion with the
fix for --no-ensure-shutdown to keep a cleaner history.
+ # Check that incompatible options error out.
+ command_fails(
+ [
+ 'pg_rewind', "--debug",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata", "-R",
+ "--no-ensure-shutdown"
+ ],
+ 'pg_rewind local with -R');
Incompatible options had better be checked within a separate perl
script? We generally do that for the other binaries.
--
Michael
Attachments:
pgrewind-dry_run-fixes.patch (text/x-diff, 2.3K) download
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index a7fd9e0cab..2cc32cd404 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -38,6 +38,7 @@ static void createBackupLabel(XLogRecPtr startpoint, TimeLineID starttli,
static void digestControlFile(ControlFileData *ControlFile, char *source,
size_t size);
+static void updateControlFile(ControlFileData *ControlFile);
static void syncTargetDirectory(void);
static void sanityChecks(void);
static void findCommonAncestorTimeline(XLogRecPtr *recptr, int *tliIndex);
@@ -101,7 +102,7 @@ main(int argc, char **argv)
{"write-recovery-conf", no_argument, NULL, 'R'},
{"source-pgdata", required_argument, NULL, 1},
{"source-server", required_argument, NULL, 2},
- {"no-ensure-shutdown", no_argument, NULL, 44},
+ {"no-ensure-shutdown", no_argument, NULL, 4},
{"version", no_argument, NULL, 'V'},
{"dry-run", no_argument, NULL, 'n'},
{"no-sync", no_argument, NULL, 'N'},
@@ -341,7 +342,7 @@ main(int argc, char **argv)
if (!rewind_needed)
{
pg_log_info("no rewind required");
- if (writerecoveryconf)
+ if (writerecoveryconf && !dry_run)
WriteRecoveryConfig(conn, datadir_target,
GenerateRecoveryConfig(conn, NULL));
exit(0);
@@ -435,13 +436,13 @@ main(int argc, char **argv)
ControlFile_new.minRecoveryPoint = endrec;
ControlFile_new.minRecoveryPointTLI = endtli;
ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY;
- update_controlfile(datadir_target, &ControlFile_new, do_sync);
+ updateControlFile(&ControlFile_new);
if (showprogress)
pg_log_info("syncing target data directory");
syncTargetDirectory();
- if (writerecoveryconf)
+ if (writerecoveryconf && !dry_run)
WriteRecoveryConfig(conn, datadir_target,
GenerateRecoveryConfig(conn, NULL));
@@ -782,6 +783,22 @@ digestControlFile(ControlFileData *ControlFile, char *src, size_t size)
checkControlFile(ControlFile);
}
+/*
+ * Update the target's control file.
+ */
+static void
+updateControlFile(ControlFileData *ControlFile)
+{
+ /* update progress report with control file size */
+ fetch_done += PG_CONTROL_FILE_SIZE;
+ progress_report(false);
+
+ if (dry_run)
+ return;
+
+ update_controlfile(datadir_target, ControlFile, do_sync);
+}
+
/*
* Sync target data directory to ensure that modifications are safely on disk.
*
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-27 12:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-30 07:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-30 17:13 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-10-02 17:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-03 03:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
@ 2019-10-03 09:43 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-04 08:37 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 46+ messages in thread
From: Alexey Kondratov @ 2019-10-03 09:43 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Paul Guo <pguo@pivotal.io>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On 03.10.2019 6:07, Michael Paquier wrote:
> On Wed, Oct 02, 2019 at 08:28:09PM +0300, Alexey Kondratov wrote:
>> I've directly followed your guess and tried to elaborate pg_rewind test
>> cases and... It seems I've caught a few bugs:
>>
>> 1) --dry-run actually wasn't completely 'dry'. It did update target
>> controlfile, which could cause repetitive pg_rewind calls to fail after
>> dry-run ones.
> I have just paid attention to this thread, but this is a bug which
> goes down to 12 actually so let's treat it independently of the rest.
> The control file was not written thanks to the safeguards in
> write_target_range() in past versions, but the recent refactoring
> around control file handling broke that promise. Another thing which
> is not completely exact is the progress reporting which should be
> reported even if the dry-run mode runs. That's less critical, but
> let's make things consistent.
I also thought about v12, though didn't check whether it's affected.
> Patch 0001 also forgot that recovery.conf should not be written either
> when no rewind is needed.
Yes, definitely, I forgot this code path, thanks.
> I have reworked your first patch as per the attached. What do you
> think about it? The part with the control file needs to go down to
> v12, and I would likely split that into two commits on HEAD: one for
> the control file and a second for the recovery.conf portion with the
> fix for --no-ensure-shutdown to keep a cleaner history.
It looks fine for me excepting the progress reporting part. It now adds
PG_CONTROL_FILE_SIZE to fetch_done. However, I cannot find that control
file is either included into filemap and fetch_size or counted during
calculate_totals(). Maybe I've missed something, but now it looks like
we report something that wasn't planned for progress reporting, doesn't it?
> + # Check that incompatible options error out.
> + command_fails(
> + [
> + 'pg_rewind', "--debug",
> + "--source-pgdata=$standby_pgdata",
> + "--target-pgdata=$master_pgdata", "-R",
> + "--no-ensure-shutdown"
> + ],
> + 'pg_rewind local with -R');
> Incompatible options had better be checked within a separate perl
> script? We generally do that for the other binaries.
Yes, it makes sense. I've reworked the patch with tests and added a
couple of extra cases.
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
Attachments:
v2-0002-Increase-pg_rewind-code-coverage.patch (text/x-patch, 6.2K) download
From 9e828e311dc7c216e5bfb1936022be4f7fd3805f Mon Sep 17 00:00:00 2001
From: Alexey Kondratov <kondratov.aleksey@gmail.com>
Date: Thu, 3 Oct 2019 12:37:26 +0300
Subject: [PATCH v2 2/2] Increase pg_rewind code coverage
Branch: pg-rewind-fixes
---
src/bin/pg_rewind/t/001_basic.pl | 2 +-
src/bin/pg_rewind/t/002_databases.pl | 2 +-
src/bin/pg_rewind/t/003_extrafiles.pl | 2 +-
src/bin/pg_rewind/t/004_pg_xlog_symlink.pl | 2 +-
src/bin/pg_rewind/t/005_same_timeline.pl | 32 +++++++++---
src/bin/pg_rewind/t/006_actions.pl | 61 ++++++++++++++++++++++
src/bin/pg_rewind/t/RewindTest.pm | 20 ++++++-
7 files changed, 107 insertions(+), 14 deletions(-)
create mode 100644 src/bin/pg_rewind/t/006_actions.pl
diff --git a/src/bin/pg_rewind/t/001_basic.pl b/src/bin/pg_rewind/t/001_basic.pl
index c3293e93df..1ba1648af6 100644
--- a/src/bin/pg_rewind/t/001_basic.pl
+++ b/src/bin/pg_rewind/t/001_basic.pl
@@ -1,7 +1,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 11;
+use Test::More tests => 13;
use FindBin;
use lib $FindBin::RealBin;
diff --git a/src/bin/pg_rewind/t/002_databases.pl b/src/bin/pg_rewind/t/002_databases.pl
index 1db534c0dc..57674ff4b3 100644
--- a/src/bin/pg_rewind/t/002_databases.pl
+++ b/src/bin/pg_rewind/t/002_databases.pl
@@ -1,7 +1,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 7;
+use Test::More tests => 9;
use FindBin;
use lib $FindBin::RealBin;
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index f4710440fc..16c92cb2d6 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -3,7 +3,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 5;
+use Test::More tests => 7;
use File::Find;
diff --git a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
index 639eeb9c91..6dabd11db6 100644
--- a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
+++ b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
@@ -14,7 +14,7 @@ if ($windows_os)
}
else
{
- plan tests => 5;
+ plan tests => 7;
}
use FindBin;
diff --git a/src/bin/pg_rewind/t/005_same_timeline.pl b/src/bin/pg_rewind/t/005_same_timeline.pl
index 40dbc44caa..089466a06b 100644
--- a/src/bin/pg_rewind/t/005_same_timeline.pl
+++ b/src/bin/pg_rewind/t/005_same_timeline.pl
@@ -1,19 +1,35 @@
+#
+# Test that running pg_rewind if the two clusters are on the same
+# timeline runs successfully.
+#
use strict;
use warnings;
use TestLib;
-use Test::More tests => 1;
+use Test::More tests => 5;
use FindBin;
use lib $FindBin::RealBin;
use RewindTest;
-# Test that running pg_rewind if the two clusters are on the same
-# timeline runs successfully.
+sub run_test
+{
+ my $test_mode = shift;
+
+ RewindTest::setup_cluster($test_mode, ['-g']);
+ RewindTest::start_master();
+ RewindTest::create_standby($test_mode);
+
+ RewindTest::promote_standby();
+
+ RewindTest::run_pg_rewind($test_mode);
+
+ RewindTest::clean_rewind_test();
+ return;
+}
+
+# Run the test in both modes.
+run_test('local');
+run_test('remote');
-RewindTest::setup_cluster();
-RewindTest::start_master();
-RewindTest::create_standby();
-RewindTest::run_pg_rewind('local');
-RewindTest::clean_rewind_test();
exit(0);
diff --git a/src/bin/pg_rewind/t/006_actions.pl b/src/bin/pg_rewind/t/006_actions.pl
new file mode 100644
index 0000000000..0667bffeae
--- /dev/null
+++ b/src/bin/pg_rewind/t/006_actions.pl
@@ -0,0 +1,61 @@
+#
+# Test incompatible options and pg_rewind internal sanity checks.
+#
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 3;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+use RewindTest;
+
+RewindTest::setup_cluster('local', ['-g']);
+RewindTest::start_master();
+RewindTest::create_standby('local');
+
+my $master_pgdata = $node_master->data_dir;
+my $standby_pgdata = $node_standby->data_dir;
+
+RewindTest::promote_standby();
+
+# Check that pg_rewind errors out if target server
+# wasn't shutdown.
+command_fails(
+ [
+ 'pg_rewind', "--debug",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata",
+ "--no-ensure-shutdown"
+ ],
+ 'pg_rewind local without target shutdown');
+
+$node_master->stop;
+
+# Check that pg_rewind errors out if source server
+# wasn't shutdown.
+command_fails(
+ [
+ 'pg_rewind', "--debug",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata",
+ "--no-ensure-shutdown"
+ ],
+ 'pg_rewind local without source shutdown');
+
+$node_standby->stop;
+
+# Check that incompatible options error out.
+command_fails(
+ [
+ 'pg_rewind', "--debug",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata", "-R",
+ "--no-ensure-shutdown"
+ ],
+ 'pg_rewind local with -R');
+
+RewindTest::clean_rewind_test();
+
+exit(0);
diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm
index c540722420..1cc88f8551 100644
--- a/src/bin/pg_rewind/t/RewindTest.pm
+++ b/src/bin/pg_rewind/t/RewindTest.pm
@@ -246,7 +246,7 @@ sub run_pg_rewind
{
# Do rewind using a local pgdata as source
- # Stop the master and be ready to perform the rewind
+ # Stop the new master and be ready to perform the rewind
$node_standby->stop;
command_ok(
[
@@ -261,6 +261,21 @@ sub run_pg_rewind
elsif ($test_mode eq "remote")
{
+ # Do rewind in dry-run mode using a remote connection as source
+ command_ok(
+ [
+ 'pg_rewind',
+ "--source-server", $standby_connstr,
+ "--target-pgdata=$master_pgdata", "-R",
+ "--debug", "--no-ensure-shutdown", "-P",
+ "--no-sync", "--dry-run"
+ ],
+ 'pg_rewind remote with --dry-run');
+
+ # Check that standby.signal hasn't been created.
+ ok(! -e "$master_pgdata/standby.signal",
+ 'standby.signal is absent');
+
# Do rewind using a remote connection as source
command_ok(
[
@@ -272,7 +287,8 @@ sub run_pg_rewind
'pg_rewind remote');
# Check that standby.signal has been created.
- ok(-e "$master_pgdata/standby.signal");
+ ok(-e "$master_pgdata/standby.signal",
+ 'standby.signal exists');
# Now, when pg_rewind apparently succeeded with minimal permissions,
# add REPLICATION privilege. So we could test that new standby
--
2.17.1
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-27 12:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-30 07:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-30 17:13 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-10-02 17:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-03 03:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-10-03 09:43 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
@ 2019-10-04 08:37 ` Michael Paquier <michael@paquier.xyz>
2019-10-04 14:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
0 siblings, 1 reply; 46+ messages in thread
From: Michael Paquier @ 2019-10-04 08:37 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Paul Guo <pguo@pivotal.io>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On Thu, Oct 03, 2019 at 12:43:37PM +0300, Alexey Kondratov wrote:
> On 03.10.2019 6:07, Michael Paquier wrote:
>> I have reworked your first patch as per the attached. What do you
>> think about it? The part with the control file needs to go down to
>> v12, and I would likely split that into two commits on HEAD: one for
>> the control file and a second for the recovery.conf portion with the
>> fix for --no-ensure-shutdown to keep a cleaner history.
>
> It looks fine for me excepting the progress reporting part. It now adds
> PG_CONTROL_FILE_SIZE to fetch_done. However, I cannot find that control file
> is either included into filemap and fetch_size or counted during
> calculate_totals(). Maybe I've missed something, but now it looks like we
> report something that wasn't planned for progress reporting, doesn't
> it?
Right. The pre-12 code actually handles that incorrecly as it assumed
that any files written through file_ops.c should be part of the
progress. So I went with the simplest solution, and backpatched this
part with 6f3823b. I have also committed the set of fixes for the new
options so as we have a better base of work than what's on HEAD
currently.
>> + # Check that incompatible options error out.
>> + command_fails(
>> + [
>> + 'pg_rewind', "--debug",
>> + "--source-pgdata=$standby_pgdata",
>> + "--target-pgdata=$master_pgdata", "-R",
>> + "--no-ensure-shutdown"
>> + ],
>> + 'pg_rewind local with -R');
>> Incompatible options had better be checked within a separate perl
>> script? We generally do that for the other binaries.
>
> Yes, it makes sense. I've reworked the patch with tests and added a couple
> of extra cases.
Regarding the tests, adding a --dry-run command is a good idea.
However I think that there is more value to automate the use of the
single user mode automatically in the tests as that's more critical
from the point of view of rewind run, and stopping the cluster with
immediate mode causes, as expected, the next --dry-run command to
fail.
Another thing is that I think that we should use -F with --single.
This makes recovery faster, and the target data folder is synced
at the end of pg_rewind anyway.
Using the long option names makes the tests easier to follow in this
case, so I have switched -R to --write-recovery-conf.
Some comments and the docs have been using some confusing wording, so
I have reworked what I found (like many "it" in a single sentence
referring different things).
+command_fails(
+ [
+ 'pg_rewind', "--debug",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata",
+ "--no-ensure-shutdown"
+ ],
+ 'pg_rewind local without source shutdown');
Regarding all the set of incompatible options, we have much more of
that after the initial option parsing so I think that we should group
all the cheap ones together. Let's tackle that as a separate patch.
We can also just check after --no-ensure-shutdown directly in
RewindTest.pm as I have switched the cluster to not be cleanly shut
down anymore to stress the automatic recovery path, and trigger that
before running pg_rewind for the local and remote mode.
Attached is an updated patch with all I found. What do you think?
--
Michael
Attachments:
pgrewind-opts-fixes.patch (text/x-diff, 4.9K) download
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index fbf454803b..42d29edd4e 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -169,12 +169,14 @@ PostgreSQL documentation
<term><option>--no-ensure-shutdown</option></term>
<listitem>
<para>
- <application>pg_rewind</application> verifies that the target server
- is cleanly shutdown before rewinding; by default, if it isn't, it
- starts the server in single-user mode to complete crash recovery.
+ <application>pg_rewind</application> requires that the target server
+ is cleanly shut down before rewinding. By default, if the target server
+ is not shut down cleanly, <application>pg_rewind</application> starts
+ the target server in single-user mode to complete crash recovery first,
+ and stops it.
By passing this option, <application>pg_rewind</application> skips
this and errors out immediately if the server is not cleanly shut
- down. Users are expected to handle the situation themselves in that
+ down. Users are expected to handle the situation themselves in that
case.
</para>
</listitem>
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index fe1468b771..875a43b219 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -270,11 +270,12 @@ main(int argc, char **argv)
pg_free(buffer);
/*
- * If the target instance was not cleanly shut down, run a single-user
- * postgres session really quickly and reload the control file to get the
- * new state. Note if no_ensure_shutdown is specified, pg_rewind won't do
- * that automatically. That means users need to do themselves in advance,
- * else pg_rewind will soon quit, see sanityChecks().
+ * If the target instance was not cleanly shut down, start and stop the
+ * target cluster once in single-user mode to enforce recovery to finish,
+ * ensuring that the cluster can be used by pg_rewind. Note that if
+ * no_ensure_shutdown is specified, pg_rewind ignores this step, and users
+ * need to make sure by themselves that the target cluster is in a clean
+ * state.
*/
if (!no_ensure_shutdown &&
ControlFile_target.state != DB_SHUTDOWNED &&
@@ -847,8 +848,12 @@ ensureCleanShutdown(const char *argv0)
if (dry_run)
return;
- /* finally run postgres in single-user mode */
- snprintf(cmd, MAXCMDLEN, "\"%s\" --single -D \"%s\" template1 < \"%s\"",
+ /*
+ * Finally run postgres in single-user mode. There is no need to use
+ * fsync here. This makes the recovery faster, and the target data folder
+ * is synced at the end anyway.
+ */
+ snprintf(cmd, MAXCMDLEN, "\"%s\" --single -F -D \"%s\" template1 < \"%s\"",
exec_path, datadir_target, DEVNULL);
if (system(cmd) != 0)
diff --git a/src/bin/pg_rewind/t/005_same_timeline.pl b/src/bin/pg_rewind/t/005_same_timeline.pl
index 40dbc44caa..df469d3939 100644
--- a/src/bin/pg_rewind/t/005_same_timeline.pl
+++ b/src/bin/pg_rewind/t/005_same_timeline.pl
@@ -1,3 +1,7 @@
+#
+# Test that running pg_rewind with the source and target clusters
+# on the same timeline runs successfully.
+#
use strict;
use warnings;
use TestLib;
diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm
index c540722420..82fa220ac8 100644
--- a/src/bin/pg_rewind/t/RewindTest.pm
+++ b/src/bin/pg_rewind/t/RewindTest.pm
@@ -227,8 +227,10 @@ sub run_pg_rewind
# Append the rewind-specific role to the connection string.
$standby_connstr = "$standby_connstr user=rewind_user";
- # Stop the master and be ready to perform the rewind
- $node_master->stop;
+ # Stop the master and be ready to perform the rewind. The cluster
+ # needs recovery to finish once, and pg_rewind makes sure that it
+ # happens automatically.
+ $node_master->stop('immediate');
# At this point, the rewind processing is ready to run.
# We now have a very simple scenario with a few diverged WAL record.
@@ -260,19 +262,21 @@ sub run_pg_rewind
}
elsif ($test_mode eq "remote")
{
-
- # Do rewind using a remote connection as source
+ # Do rewind using a remote connection as source, generating
+ # recovery configuration automatically.
command_ok(
[
'pg_rewind', "--debug",
"--source-server", $standby_connstr,
- "--target-pgdata=$master_pgdata", "-R",
- "--no-sync"
+ "--target-pgdata=$master_pgdata", "--no-sync",
+ "--write-recovery-conf"
],
'pg_rewind remote');
- # Check that standby.signal has been created.
- ok(-e "$master_pgdata/standby.signal");
+ # Check that standby.signal is here as recovery configuration
+ # was requested.
+ ok( -e "$master_pgdata/standby.signal",
+ 'standby.signal created after pg_rewind');
# Now, when pg_rewind apparently succeeded with minimal permissions,
# add REPLICATION privilege. So we could test that new standby
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-27 12:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-30 07:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-30 17:13 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-10-02 17:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-03 03:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-10-03 09:43 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-04 08:37 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
@ 2019-10-04 14:21 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-07 01:06 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 46+ messages in thread
From: Alexey Kondratov @ 2019-10-04 14:21 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Paul Guo <pguo@pivotal.io>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On 04.10.2019 11:37, Michael Paquier wrote:
> On Thu, Oct 03, 2019 at 12:43:37PM +0300, Alexey Kondratov wrote:
>> On 03.10.2019 6:07, Michael Paquier wrote:
>>> I have reworked your first patch as per the attached. What do you
>>> think about it? The part with the control file needs to go down to
>>> v12, and I would likely split that into two commits on HEAD: one for
>>> the control file and a second for the recovery.conf portion with the
>>> fix for --no-ensure-shutdown to keep a cleaner history.
>> It looks fine for me excepting the progress reporting part. It now adds
>> PG_CONTROL_FILE_SIZE to fetch_done. However, I cannot find that control file
>> is either included into filemap and fetch_size or counted during
>> calculate_totals(). Maybe I've missed something, but now it looks like we
>> report something that wasn't planned for progress reporting, doesn't
>> it?
> Right. The pre-12 code actually handles that incorrecly as it assumed
> that any files written through file_ops.c should be part of the
> progress. So I went with the simplest solution, and backpatched this
> part with 6f3823b. I have also committed the set of fixes for the new
> options so as we have a better base of work than what's on HEAD
> currently.
Great, thanks.
>
> Regarding the tests, adding a --dry-run command is a good idea.
> However I think that there is more value to automate the use of the
> single user mode automatically in the tests as that's more critical
> from the point of view of rewind run, and stopping the cluster with
> immediate mode causes, as expected, the next --dry-run command to
> fail.
>
> Another thing is that I think that we should use -F with --single.
> This makes recovery faster, and the target data folder is synced
> at the end of pg_rewind anyway.
>
> Using the long option names makes the tests easier to follow in this
> case, so I have switched -R to --write-recovery-conf.
>
> Some comments and the docs have been using some confusing wording, so
> I have reworked what I found (like many "it" in a single sentence
> referring different things).
I agree with all the points. Shutting down target server using
'immediate' mode is a good way to test ensureCleanShutdown automatically.
> Regarding all the set of incompatible options, we have much more of
> that after the initial option parsing so I think that we should group
> all the cheap ones together. Let's tackle that as a separate patch.
> We can also just check after --no-ensure-shutdown directly in
> RewindTest.pm as I have switched the cluster to not be cleanly shut
> down anymore to stress the automatic recovery path, and trigger that
> before running pg_rewind for the local and remote mode.
>
> Attached is an updated patch with all I found. What do you think?
I've checked your patch, but it seems that it cannot be applied as is,
since it e.g. adds a comment to 005_same_timeline.pl without actually
changing the test. So I've slightly modified your patch and tried to fit
both dry-run and ensureCleanShutdown testing together. It works just
fine and fails immediately if any of recent fixes is reverted. I still
think that dry-run testing is worth adding, since it helped to catch
this v12 refactoring issue, but feel free to throw it way if it isn't
commitable right now, of course.
As for incompatible options and sanity checks testing, yes, I agree that
it is a matter of different patch. I attached it as a separate WIP patch
just for history. Maybe I will try to gather more cases there later.
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
Attachments: download all (2 files, .tar.gz)
v3-0001-Improve-functionality-docs-and-tests-of-R-no-ensu.patch (text/x-patch, 9.5K) download
From 6e5667edcad6b037004288635a7ae0eda40d4262 Mon Sep 17 00:00:00 2001
From: Alexey Kondratov <kondratov.aleksey@gmail.com>
Date: Fri, 4 Oct 2019 17:14:12 +0300
Subject: [PATCH v3 1/2] Improve functionality, docs and tests of -R,
--no-ensure-shutdown and --dry-run options
Branch: pg-rewind-fixes
---
doc/src/sgml/ref/pg_rewind.sgml | 10 +--
src/bin/pg_rewind/pg_rewind.c | 19 +++---
src/bin/pg_rewind/t/001_basic.pl | 2 +-
src/bin/pg_rewind/t/002_databases.pl | 2 +-
src/bin/pg_rewind/t/003_extrafiles.pl | 2 +-
src/bin/pg_rewind/t/004_pg_xlog_symlink.pl | 2 +-
src/bin/pg_rewind/t/005_same_timeline.pl | 32 +++++++---
src/bin/pg_rewind/t/RewindTest.pm | 71 +++++++++++++++++-----
8 files changed, 103 insertions(+), 37 deletions(-)
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index fbf454803b..42d29edd4e 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -169,12 +169,14 @@ PostgreSQL documentation
<term><option>--no-ensure-shutdown</option></term>
<listitem>
<para>
- <application>pg_rewind</application> verifies that the target server
- is cleanly shutdown before rewinding; by default, if it isn't, it
- starts the server in single-user mode to complete crash recovery.
+ <application>pg_rewind</application> requires that the target server
+ is cleanly shut down before rewinding. By default, if the target server
+ is not shut down cleanly, <application>pg_rewind</application> starts
+ the target server in single-user mode to complete crash recovery first,
+ and stops it.
By passing this option, <application>pg_rewind</application> skips
this and errors out immediately if the server is not cleanly shut
- down. Users are expected to handle the situation themselves in that
+ down. Users are expected to handle the situation themselves in that
case.
</para>
</listitem>
diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c
index fe1468b771..875a43b219 100644
--- a/src/bin/pg_rewind/pg_rewind.c
+++ b/src/bin/pg_rewind/pg_rewind.c
@@ -270,11 +270,12 @@ main(int argc, char **argv)
pg_free(buffer);
/*
- * If the target instance was not cleanly shut down, run a single-user
- * postgres session really quickly and reload the control file to get the
- * new state. Note if no_ensure_shutdown is specified, pg_rewind won't do
- * that automatically. That means users need to do themselves in advance,
- * else pg_rewind will soon quit, see sanityChecks().
+ * If the target instance was not cleanly shut down, start and stop the
+ * target cluster once in single-user mode to enforce recovery to finish,
+ * ensuring that the cluster can be used by pg_rewind. Note that if
+ * no_ensure_shutdown is specified, pg_rewind ignores this step, and users
+ * need to make sure by themselves that the target cluster is in a clean
+ * state.
*/
if (!no_ensure_shutdown &&
ControlFile_target.state != DB_SHUTDOWNED &&
@@ -847,8 +848,12 @@ ensureCleanShutdown(const char *argv0)
if (dry_run)
return;
- /* finally run postgres in single-user mode */
- snprintf(cmd, MAXCMDLEN, "\"%s\" --single -D \"%s\" template1 < \"%s\"",
+ /*
+ * Finally run postgres in single-user mode. There is no need to use
+ * fsync here. This makes the recovery faster, and the target data folder
+ * is synced at the end anyway.
+ */
+ snprintf(cmd, MAXCMDLEN, "\"%s\" --single -F -D \"%s\" template1 < \"%s\"",
exec_path, datadir_target, DEVNULL);
if (system(cmd) != 0)
diff --git a/src/bin/pg_rewind/t/001_basic.pl b/src/bin/pg_rewind/t/001_basic.pl
index c3293e93df..a1659460ec 100644
--- a/src/bin/pg_rewind/t/001_basic.pl
+++ b/src/bin/pg_rewind/t/001_basic.pl
@@ -1,7 +1,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 11;
+use Test::More tests => 14;
use FindBin;
use lib $FindBin::RealBin;
diff --git a/src/bin/pg_rewind/t/002_databases.pl b/src/bin/pg_rewind/t/002_databases.pl
index 1db534c0dc..921c4434f5 100644
--- a/src/bin/pg_rewind/t/002_databases.pl
+++ b/src/bin/pg_rewind/t/002_databases.pl
@@ -1,7 +1,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 7;
+use Test::More tests => 10;
use FindBin;
use lib $FindBin::RealBin;
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index f4710440fc..bce5b47148 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -3,7 +3,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 5;
+use Test::More tests => 8;
use File::Find;
diff --git a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
index 639eeb9c91..a501be8f78 100644
--- a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
+++ b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
@@ -14,7 +14,7 @@ if ($windows_os)
}
else
{
- plan tests => 5;
+ plan tests => 8;
}
use FindBin;
diff --git a/src/bin/pg_rewind/t/005_same_timeline.pl b/src/bin/pg_rewind/t/005_same_timeline.pl
index 40dbc44caa..1d96740a55 100644
--- a/src/bin/pg_rewind/t/005_same_timeline.pl
+++ b/src/bin/pg_rewind/t/005_same_timeline.pl
@@ -1,19 +1,35 @@
+#
+# Test that running pg_rewind with the source and target clusters
+# on the same timeline runs successfully.
+#
use strict;
use warnings;
use TestLib;
-use Test::More tests => 1;
+use Test::More tests => 6;
use FindBin;
use lib $FindBin::RealBin;
use RewindTest;
-# Test that running pg_rewind if the two clusters are on the same
-# timeline runs successfully.
+sub run_test
+{
+ my $test_mode = shift;
+
+ RewindTest::setup_cluster($test_mode, ['-g']);
+ RewindTest::start_master();
+ RewindTest::create_standby($test_mode);
+
+ RewindTest::promote_standby();
+
+ RewindTest::run_pg_rewind($test_mode);
+
+ RewindTest::clean_rewind_test();
+ return;
+}
+
+# Run the test in both modes.
+run_test('local');
+run_test('remote');
-RewindTest::setup_cluster();
-RewindTest::start_master();
-RewindTest::create_standby();
-RewindTest::run_pg_rewind('local');
-RewindTest::clean_rewind_test();
exit(0);
diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm
index c540722420..9292092b1a 100644
--- a/src/bin/pg_rewind/t/RewindTest.pm
+++ b/src/bin/pg_rewind/t/RewindTest.pm
@@ -227,9 +227,57 @@ sub run_pg_rewind
# Append the rewind-specific role to the connection string.
$standby_connstr = "$standby_connstr user=rewind_user";
- # Stop the master and be ready to perform the rewind
+ # Stop the master cleanly to check dry-run mode.
$node_master->stop;
+ # First run pg_rewind in dry-run mode
+ if ($test_mode eq "local")
+ {
+
+ # Stop the new master and be ready to perform the rewind
+ # Do rewind in dry-run mode using a local pgdata as source
+ $node_standby->stop;
+ command_ok(
+ [
+ 'pg_rewind',
+ "--debug", "--no-sync", "--dry-run",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata",
+ "--no-ensure-shutdown"
+ ],
+ 'pg_rewind local with --dry-run');
+ }
+ elsif ($test_mode eq "remote")
+ {
+
+ # Do rewind in dry-run mode using a remote connection as source
+ command_ok(
+ [
+ 'pg_rewind',
+ "--debug", "--no-sync", "--dry-run",
+ "--source-server", $standby_connstr,
+ "--target-pgdata=$master_pgdata",
+ "--write-recovery-conf", "--progress"
+ ],
+ 'pg_rewind remote with --dry-run');
+
+ # Check that standby.signal hasn't been created.
+ ok(! -e "$master_pgdata/standby.signal",
+ 'standby.signal is absent');
+ }
+ else
+ {
+
+ # Cannot come here normally
+ croak("Incorrect test mode specified");
+ }
+
+ # Start and stop the master and be ready to perform the rewind.
+ # The cluster needs recovery to finish once, and pg_rewind makes
+ # sure that it happens automatically.
+ $node_master->start;
+ $node_master->stop('immediate');
+
# At this point, the rewind processing is ready to run.
# We now have a very simple scenario with a few diverged WAL record.
# The real testing begins really now with a bifurcation of the possible
@@ -246,8 +294,6 @@ sub run_pg_rewind
{
# Do rewind using a local pgdata as source
- # Stop the master and be ready to perform the rewind
- $node_standby->stop;
command_ok(
[
'pg_rewind',
@@ -261,18 +307,21 @@ sub run_pg_rewind
elsif ($test_mode eq "remote")
{
- # Do rewind using a remote connection as source
+ # Do rewind using a remote connection as source, generating
+ # recovery configuration automatically.
command_ok(
[
'pg_rewind', "--debug",
"--source-server", $standby_connstr,
- "--target-pgdata=$master_pgdata", "-R",
- "--no-sync"
+ "--target-pgdata=$master_pgdata", "--no-sync",
+ "--write-recovery-conf"
],
'pg_rewind remote');
- # Check that standby.signal has been created.
- ok(-e "$master_pgdata/standby.signal");
+ # Check that standby.signal is here as recovery configuration
+ # was requested.
+ ok( -e "$master_pgdata/standby.signal",
+ 'standby.signal created after pg_rewind');
# Now, when pg_rewind apparently succeeded with minimal permissions,
# add REPLICATION privilege. So we could test that new standby
@@ -280,12 +329,6 @@ sub run_pg_rewind
$node_standby->safe_psql('postgres',
"ALTER ROLE rewind_user WITH REPLICATION;");
}
- else
- {
-
- # Cannot come here normally
- croak("Incorrect test mode specified");
- }
# Now move back postgresql.conf with old settings
move(
base-commit: 6837632b758e0470a2692d9f8303e8aebd6fbd8f
--
2.17.1
v3-0002-WIP-increase-pg_rewind-test-coverage.patch (text/x-patch, 2.1K) download
From 50c4ac745354c38ece3a805a4882eb1b2895a3fe Mon Sep 17 00:00:00 2001
From: Alexey Kondratov <kondratov.aleksey@gmail.com>
Date: Fri, 4 Oct 2019 17:14:53 +0300
Subject: [PATCH v3 2/2] WIP: increase pg_rewind test coverage
Branch: pg-rewind-fixes
---
src/bin/pg_rewind/t/006_actions.pl | 62 ++++++++++++++++++++++++++++++
1 file changed, 62 insertions(+)
create mode 100644 src/bin/pg_rewind/t/006_actions.pl
diff --git a/src/bin/pg_rewind/t/006_actions.pl b/src/bin/pg_rewind/t/006_actions.pl
new file mode 100644
index 0000000000..439074ee72
--- /dev/null
+++ b/src/bin/pg_rewind/t/006_actions.pl
@@ -0,0 +1,62 @@
+#
+# Test incompatible options and pg_rewind internal sanity checks.
+#
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 3;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+use RewindTest;
+
+RewindTest::setup_cluster('local', ['-g']);
+RewindTest::start_master();
+RewindTest::create_standby('local');
+
+my $master_pgdata = $node_master->data_dir;
+my $standby_pgdata = $node_standby->data_dir;
+
+RewindTest::promote_standby();
+
+# Check that pg_rewind errors out if target server
+# wasn't shutdown.
+command_fails(
+ [
+ 'pg_rewind', "--debug",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata",
+ "--no-ensure-shutdown"
+ ],
+ 'pg_rewind local without target shutdown');
+
+$node_master->stop;
+
+# Check that pg_rewind errors out if source server
+# wasn't shutdown.
+command_fails(
+ [
+ 'pg_rewind', "--debug",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata",
+ "--no-ensure-shutdown"
+ ],
+ 'pg_rewind local without source shutdown');
+
+$node_standby->stop;
+
+# Check that incompatible options error out.
+command_fails(
+ [
+ 'pg_rewind', "--debug",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata",
+ "--write-recovery-conf",
+ "--no-ensure-shutdown"
+ ],
+ 'pg_rewind local with --write-recovery-conf');
+
+RewindTest::clean_rewind_test();
+
+exit(0);
--
2.17.1
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-27 12:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-30 07:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-30 17:13 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-10-02 17:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-03 03:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-10-03 09:43 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-04 08:37 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-10-04 14:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
@ 2019-10-07 01:06 ` Michael Paquier <michael@paquier.xyz>
2019-10-07 12:31 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
0 siblings, 1 reply; 46+ messages in thread
From: Michael Paquier @ 2019-10-07 01:06 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Paul Guo <pguo@pivotal.io>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On Fri, Oct 04, 2019 at 05:21:25PM +0300, Alexey Kondratov wrote:
> I've checked your patch, but it seems that it cannot be applied as is, since
> it e.g. adds a comment to 005_same_timeline.pl without actually changing the
> test. So I've slightly modified your patch and tried to fit both dry-run and
> ensureCleanShutdown testing together. It works just fine and fails
> immediately if any of recent fixes is reverted. I still think that dry-run
> testing is worth adding, since it helped to catch this v12 refactoring
> issue, but feel free to throw it way if it isn't commitable right now, of
> course.
I can guarantee the last patch I sent can be applied on top of HEAD:
https://www.postgresql.org/message-id/20191004083721.GA1829@paquier.xyz
It would be nice to add the --dry-run part, though I think that we
could just make that part of one of the existing tests, and stop the
target server first (got to think about that part, please see below).
> As for incompatible options and sanity checks testing, yes, I agree that it
> is a matter of different patch. I attached it as a separate WIP patch just
> for history. Maybe I will try to gather more cases there later.
Thanks. I have applied the first patch for the various improvements
around --no-ensure-shutdown.
Regarding the rest, I have hacked my way through as per the attached.
The previous set of patches did the following, which looked either
overkill or not necessary:
- Why running test 005 with the remote mode?
- --dry-run coverage is basically the same with the local and remote
modes, so it seems like a waste of resource to run it for all the
tests and all the modes. I tend to think that this would live better
as part of another existing test, only running for say the local mode.
It is also possible to group all your tests from patch 2 and
006_actions.pl in this area.
- There is no need for the script checking for options combinations to
initialize a data folder. It is important to design the tests to be
cheap and meaningful.
Patch v3-0002 also had a test to make sure that the source server is
shut down cleanly before using it. I have included that part as
well, as the flow feels right.
So, Alexey, what do you think?
--
Michael
Attachments:
rewind-new-tests.patch (text/x-diff, 3.7K) download
diff --git a/src/bin/pg_rewind/t/001_basic.pl b/src/bin/pg_rewind/t/001_basic.pl
index c3293e93df..645dc24578 100644
--- a/src/bin/pg_rewind/t/001_basic.pl
+++ b/src/bin/pg_rewind/t/001_basic.pl
@@ -1,7 +1,7 @@
use strict;
use warnings;
use TestLib;
-use Test::More tests => 11;
+use Test::More tests => 15;
use FindBin;
use lib $FindBin::RealBin;
@@ -66,6 +66,76 @@ sub run_test
master_psql("DELETE FROM tail_tbl WHERE id > 10");
master_psql("VACUUM tail_tbl");
+ # Before running pg_rewind, do a couple of extra tests with several
+ # option combinations. As the code paths taken by those tests
+ # does not change for the "local" and "remote" modes, just run them
+ # in "local" mode for simplicity's sake.
+ if ($test_mode eq 'local')
+ {
+ my $master_pgdata = $node_master->data_dir;
+ my $standby_pgdata = $node_standby->data_dir;
+
+ # First check that pg_rewind fails if the target cluster is
+ # not stopped as it fails to start up for the forced recovery
+ # step.
+ command_fails(
+ [
+ 'pg_rewind',
+ "--debug",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata",
+ "--no-sync"
+ ],
+ 'pg_rewind with running target');
+
+ # Again with --no-ensure-shutdown, which should equally fail.
+ # This time pg_rewind complains without attempting to perform
+ # recovery once.
+ command_fails(
+ [
+ 'pg_rewind',
+ "--debug",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata",
+ "--no-sync",
+ '--no-ensure-shutdown'
+ ],
+ 'pg_rewind --no-ensure-shutdown with running target');
+
+ # Stop the target, and attempt to run with a local source
+ # still running. This fail as pg_rewind requires to have
+ # a source cleanly stopped.
+ $node_master->stop;
+ command_fails(
+ [
+ 'pg_rewind',
+ "--debug",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata",
+ "--no-sync",
+ '--no-ensure-shutdown'
+ ],
+ 'pg_rewind with running source');
+
+ # Stop the target cluster cleanly, and run again pg_rewind
+ # with --dry-run mode. If anything gets generated in the data
+ # folder, the follow-up run of pg_rewind will most likely fail,
+ # so keep this test as the last one of this subset.
+ $node_standby->stop;
+ command_ok(
+ [
+ 'pg_rewind', "--debug",
+ "--source-pgdata=$standby_pgdata",
+ "--target-pgdata=$master_pgdata",
+ "--no-sync", '--dry-run'
+ ],
+ 'pg_rewind --dry-run with running target');
+
+ # Both clusters need to be alive moving forward.
+ $node_standby->start;
+ $node_master->start;
+ }
+
RewindTest::run_pg_rewind($test_mode);
check_query(
diff --git a/src/bin/pg_rewind/t/006_options.pl b/src/bin/pg_rewind/t/006_options.pl
new file mode 100644
index 0000000000..6b8e2ad765
--- /dev/null
+++ b/src/bin/pg_rewind/t/006_options.pl
@@ -0,0 +1,38 @@
+#
+# Test checking options of pg_rewind.
+#
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 12;
+
+program_help_ok('pg_rewind');
+program_version_ok('pg_rewind');
+program_options_handling_ok('pg_rewind');
+
+my $primary_pgdata = TestLib::tempdir;
+my $standby_pgdata = TestLib::tempdir;
+command_fails(
+ [
+ 'pg_rewind', '--target-pgdata',
+ $primary_pgdata, '--source-pgdata',
+ $standby_pgdata, 'extra_arg1'
+ ],
+ 'too many arguments');
+command_fails([ 'pg_rewind', '--target-pgdata', $primary_pgdata ],
+ 'no source specified');
+command_fails(
+ [
+ 'pg_rewind', '--target-pgdata',
+ $primary_pgdata, '--source-pgdata',
+ $standby_pgdata, '--source-server',
+ 'popo'
+ ],
+ 'both remote and local sources specified');
+command_fails(
+ [
+ 'pg_rewind', '--target-pgdata',
+ $primary_pgdata, '--write-recovery-conf',
+ '--source-pgdata', $standby_pgdata
+ ],
+ 'no remote source with --write-recovery-conf');
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-27 12:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-30 07:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-30 17:13 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-10-02 17:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-03 03:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-10-03 09:43 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-04 08:37 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-10-04 14:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-07 01:06 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
@ 2019-10-07 12:31 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-08 02:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 46+ messages in thread
From: Alexey Kondratov @ 2019-10-07 12:31 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Paul Guo <pguo@pivotal.io>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On 07.10.2019 4:06, Michael Paquier wrote:
> On Fri, Oct 04, 2019 at 05:21:25PM +0300, Alexey Kondratov wrote:
>> I've checked your patch, but it seems that it cannot be applied as is, since
>> it e.g. adds a comment to 005_same_timeline.pl without actually changing the
>> test. So I've slightly modified your patch and tried to fit both dry-run and
>> ensureCleanShutdown testing together. It works just fine and fails
>> immediately if any of recent fixes is reverted. I still think that dry-run
>> testing is worth adding, since it helped to catch this v12 refactoring
>> issue, but feel free to throw it way if it isn't commitable right now, of
>> course.
> I can guarantee the last patch I sent can be applied on top of HEAD:
> https://www.postgresql.org/message-id/20191004083721.GA1829@paquier.xyz
Yes, it did, but my comment was about these lines:
diff --git a/src/bin/pg_rewind/t/005_same_timeline.pl
b/src/bin/pg_rewind/t/005_same_timeline.pl
index 40dbc44caa..df469d3939 100644
inline patch 1: src/bin/pg_rewind/t/005_same_timeline.pl (+4/-0 lines) download inline-1.patch
--- a/src/bin/pg_rewind/t/005_same_timeline.pl
+++ b/src/bin/pg_rewind/t/005_same_timeline.pl
@@ -1,3 +1,7 @@
+#
+# Test that running pg_rewind with the source and target clusters
+# on the same timeline runs successfully.
+#
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-27 12:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-30 07:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-30 17:13 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-10-02 17:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-03 03:07 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-10-03 09:43 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-04 08:37 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-10-04 14:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-07 01:06 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-10-07 12:31 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alexey Kondratov <a.kondratov@postgrespro.ru>
@ 2019-10-08 02:51 ` Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 46+ messages in thread
From: Michael Paquier @ 2019-10-08 02:51 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Paul Guo <pguo@pivotal.io>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On Mon, Oct 07, 2019 at 03:31:45PM +0300, Alexey Kondratov wrote:
> On 07.10.2019 4:06, Michael Paquier wrote:
>> - --dry-run coverage is basically the same with the local and remote
>> modes, so it seems like a waste of resource to run it for all the
>> tests and all the modes.
>
> My point was to test --dry-run + --write-recover-conf in remote, since the
> last one may cause recovery configuration write without doing any actual
> work, due to some wrong refactoring for example.
Yes, that's possible. I agree that it would be nice to have an extra
test for that, still I would avoid making that run in all the tests.
>> Patch v3-0002 also had a test to make sure that the source server is
>> shut down cleanly before using it. I have included that part as
>> well, as the flow feels right.
>>
>> So, Alexey, what do you think?
>
> It looks good for me. Two minor remarks:
>
> + # option combinations. As the code paths taken by those tests
> + # does not change for the "local" and "remote" modes, just run them
>
> I am far from being fluent in English, but should it be 'do not change'
> instead?
That was wrong, fixed.
> +command_fails(
> + [
> + 'pg_rewind', '--target-pgdata',
> + $primary_pgdata, '--source-pgdata',
> + $standby_pgdata, 'extra_arg1'
> + ],
>
> Here and below I would prefer traditional options ordering "'--key',
> 'value'". It should be easier to recognizefrom the reader perspective:
While I agree with you, the perl indentation we use has formatted the
code this way. There is also an argument for keeping it at the end
for clarity (I recall that Windows also requires extra args to be
last when parsing options). Anyway, I have used a trick by adding
--debug to reach command, which is still useful, so the order of the
options is better at the end.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 14:05 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-09-27 19:52 ` Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 0 replies; 46+ messages in thread
From: Alvaro Herrera @ 2019-09-27 19:52 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>; Alexey Kondratov <a.kondratov@postgrespro.ru>
On 2019-Sep-27, Paul Guo wrote:
> I'm using --no-ensure-shutdown in the new version unless there are better
> suggestions.
That sounds sufficiently good. I pushed this patch, after fixing a few
smallish problems, such as an assertion failure because of the
terminating \n in the error message when "postgres --single" fails
(which I tested by introducing a typo in the command). I also removed
the short option, because I doubt that this option is useful enough to
warrant using up such an important shorthand (Maybe if it had been
-\ or -% or -& I would have let it through, since I doubt anybody would
have wanted to use those for anything else). But if somebody disagrees,
they can send a patch to restore it, and we can then discuss the merits
of individual chars to use.
I also added quotes to DEVNULL, because we do that everywhere. Maybe
there exists a system somewhere that requires this ... !!??
Finally, I split out the command in the error message in case it fails.
Thanks.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-07-01 15:57 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-07-02 05:54 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
1 sibling, 1 reply; 46+ messages in thread
From: Alvaro Herrera @ 2019-07-01 15:57 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers@postgresql.org, Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On 2019-Apr-19, Paul Guo wrote:
> The below patch runs single mode Postgres if needed to make sure the target
> is cleanly shutdown. A new option is added (off by default).
> v2-0001-Ensure-target-clean-shutdown-at-beginning-of-pg_r.patch
Why do we need an option for this? Is there a reason not to do this
unconditionally?
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-07-01 15:57 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2019-07-02 05:54 ` Paul Guo <pguo@pivotal.io>
0 siblings, 0 replies; 46+ messages in thread
From: Paul Guo @ 2019-07-02 05:54 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On Tue, Jul 2, 2019 at 12:35 AM Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:
> On 2019-Apr-19, Paul Guo wrote:
>
> > The below patch runs single mode Postgres if needed to make sure the
> target
> > is cleanly shutdown. A new option is added (off by default).
> > v2-0001-Ensure-target-clean-shutdown-at-beginning-of-pg_r.patch
>
> Why do we need an option for this? Is there a reason not to do this
> unconditionally?
>
There is concern about this (see previous emails in this thread). On
greenplum (MPP DB based on Postgres),
we unconditionally do this. I'm not sure about usually how Postgres users
do this when there is an unclean shutdown,
but providing an option seem to be safer to avoid breaking existing
script/service whatever. If many people
think this option is unnecessary, I'm fine to remove the option and keep
the code logic.
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
@ 2019-09-30 17:15 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-10-01 02:08 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
1 sibling, 1 reply; 46+ messages in thread
From: Alvaro Herrera @ 2019-09-30 17:15 UTC (permalink / raw)
To: Paul Guo <pguo@pivotal.io>; +Cc: pgsql-hackers@postgresql.org, Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
On 2019-Mar-19, Paul Guo wrote:
> Hello, Postgres hackers,
>
> Please see the attached patches.
BTW in the future if you have two separate patches, please post them in
separate threads and use separate commitfest items for each, even if
they have minor conflicts.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-09-30 17:15 ` Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2019-10-01 02:08 ` Paul Guo <pguo@pivotal.io>
0 siblings, 0 replies; 46+ messages in thread
From: Paul Guo @ 2019-10-01 02:08 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: pgsql-hackers; Jimmy Yih <jyih@pivotal.io>; Ashwin Agrawal <aagrawal@pivotal.io>
>
> BTW in the future if you have two separate patches, please post them in
> separate threads and use separate commitfest items for each, even if
> they have minor conflicts.
>
Sure. Thanks.
^ permalink raw reply [nested|flat] 46+ messages in thread
end of thread, other threads:[~2019-10-08 02:51 UTC | newest]
Thread overview: 46+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-03-19 06:09 Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown) Paul Guo <pguo@pivotal.io>
2019-03-19 06:18 ` Michael Paquier <michael@paquier.xyz>
2019-03-20 04:48 ` Paul Guo <pguo@pivotal.io>
2019-03-20 05:20 ` Michael Paquier <michael@paquier.xyz>
2019-03-20 05:23 ` Paul Guo <pguo@pivotal.io>
2019-04-19 03:40 ` Paul Guo <pguo@pivotal.io>
2019-07-01 11:34 ` Thomas Munro <thomas.munro@gmail.com>
2019-07-02 05:46 ` Paul Guo <pguo@pivotal.io>
2019-07-08 22:54 ` Thomas Munro <thomas.munro@gmail.com>
2019-07-09 14:48 ` Paul Guo <pguo@pivotal.io>
2019-07-10 07:28 ` Michael Paquier <michael@paquier.xyz>
2019-07-15 08:52 ` Paul Guo <pguo@pivotal.io>
2019-09-03 22:39 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-05 07:41 ` Paul Guo <pguo@pivotal.io>
2019-09-05 13:23 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-09 14:18 ` Paul Guo <pguo@pivotal.io>
2019-09-10 17:52 ` Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
2019-09-19 13:21 ` Paul Guo <pguo@pivotal.io>
2019-09-20 07:33 ` Paul Guo <pguo@pivotal.io>
2019-09-24 20:12 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 02:03 ` Paul Guo <pguo@pivotal.io>
2019-09-25 17:48 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-25 19:26 ` Laurenz Albe <laurenz.albe@cybertec.at>
2019-09-25 20:22 ` a.kondratov@postgrespro.ru
2019-09-26 14:09 ` Paul Guo <pguo@pivotal.io>
2019-09-26 14:05 ` Paul Guo <pguo@pivotal.io>
2019-09-26 14:51 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-27 03:27 ` Paul Guo <pguo@pivotal.io>
2019-09-27 12:18 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-27 14:28 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-30 07:07 ` Paul Guo <pguo@pivotal.io>
2019-09-30 08:51 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-30 17:13 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-10-02 17:28 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-03 03:07 ` Michael Paquier <michael@paquier.xyz>
2019-10-03 09:43 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-04 08:37 ` Michael Paquier <michael@paquier.xyz>
2019-10-04 14:21 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-07 01:06 ` Michael Paquier <michael@paquier.xyz>
2019-10-07 12:31 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-08 02:51 ` Michael Paquier <michael@paquier.xyz>
2019-09-27 19:52 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-07-01 15:57 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-07-02 05:54 ` Paul Guo <pguo@pivotal.io>
2019-09-30 17:15 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-10-01 02:08 ` Paul Guo <pguo@pivotal.io>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox