From: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
To: Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Cc: Sergei Kornilov <sk@zsrv.org>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: base backup client as auxiliary backend process
Date: Sat, 11 Jan 2020 10:52:30 +0100
Message-ID: <4c79f835-b443-7b02-9097-aa1a40f46be9@2ndquadrant.com> (raw)
In-Reply-To: <CA+fd4k4BcYrYucNfTnK-CQX3+jsG+PRPEhHAUSo-W4P0Lec57A@mail.gmail.com>
References: <61b8d18d-c922-ac99-b990-a31ba63cdcbb@2ndquadrant.com>
<62e860f2-ae4b-67a1-b65d-04c3d1bf6aff@2ndquadrant.com>
<20190918083110.GB22307@paquier.xyz>
<3f197f5e-9f55-bef3-adc7-2b44c8f8a74f@2ndquadrant.com>
<20191107041630.GK1768@paquier.xyz>
<4792e456-d75f-8e6a-2d47-34b8f78c266f@2ndquadrant.com>
<116725031573825947@iva1-ef4837f8671e.qloud-c.yandex.net>
<8d8c851a-c951-f98e-22c2-f62621c32d3b@2ndquadrant.com>
<CA+fd4k4BcYrYucNfTnK-CQX3+jsG+PRPEhHAUSo-W4P0Lec57A@mail.gmail.com>
On 2020-01-10 04:32, Masahiko Sawada wrote:
> I agreed that these patches are useful on its own and 0001 patch and
committed 0001
> 0002 patch look good to me. For 0003 patch,
>
> + linkend="guc-primary-slot-name"/>. Otherwise, the WAL receiver may use
> + a temporary replication slot (determined by <xref
> + linkend="guc-wal-receiver-create-temp-slot"/>), but these are not shown
> + here.
>
> I think it's better to show the temporary slot name on
> pg_stat_wal_receiver view. Otherwise user would have no idea about
> what wal receiver is using the temporary slot.
Makes sense. It makes the code a bit more fiddly, but it seems worth
it. New patches attached.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:download all (2 files, .tar.gz)v2-0001-Expose-PQbackendPID-through-walreceiver-API.patch (text/plain, 3.1K) download
From 2a089e6bd34b04e17f7b2918057d8e8eb04c117f Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Sat, 11 Jan 2020 10:22:08 +0100
Subject: [PATCH v2 1/2] Expose PQbackendPID() through walreceiver API
This will be used by a subsequent patch.
---
.../replication/libpqwalreceiver/libpqwalreceiver.c | 11 +++++++++++
src/include/replication/walreceiver.h | 4 ++++
2 files changed, 15 insertions(+)
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.cindex 658af71fec..b731d3fd04 100644--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c@@ -74,6 +74,7 @@ static char *libpqrcv_create_slot(WalReceiverConn *conn,
bool temporary,
CRSSnapshotAction snapshot_action,
XLogRecPtr *lsn);
+static pid_t libpqrcv_get_backend_pid(WalReceiverConn *conn);
static WalRcvExecResult *libpqrcv_exec(WalReceiverConn *conn,
const char *query,
const int nRetTypes,
@@ -93,6 +94,7 @@ static WalReceiverFunctionsType PQWalReceiverFunctions = {
libpqrcv_receive,
libpqrcv_send,
libpqrcv_create_slot,
+ libpqrcv_get_backend_pid,
libpqrcv_exec,
libpqrcv_disconnect
};
@@ -858,6 +860,15 @@ libpqrcv_create_slot(WalReceiverConn *conn, const char *slotname,
return snapshot;
}
+/*+ * Return PID of remote backend process.+ */+static pid_t+libpqrcv_get_backend_pid(WalReceiverConn *conn)+{+ return PQbackendPID(conn->streamConn);+}+
/*
* Convert tuple query result to tuplestore.
*/
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.hindex a276237477..172cfa2862 100644--- a/src/include/replication/walreceiver.h+++ b/src/include/replication/walreceiver.h@@ -226,6 +226,7 @@ typedef char *(*walrcv_create_slot_fn) (WalReceiverConn *conn,
const char *slotname, bool temporary,
CRSSnapshotAction snapshot_action,
XLogRecPtr *lsn);
+typedef pid_t (*walrcv_get_backend_pid_fn) (WalReceiverConn *conn);
typedef WalRcvExecResult *(*walrcv_exec_fn) (WalReceiverConn *conn,
const char *query,
const int nRetTypes,
@@ -246,6 +247,7 @@ typedef struct WalReceiverFunctionsType
walrcv_receive_fn walrcv_receive;
walrcv_send_fn walrcv_send;
walrcv_create_slot_fn walrcv_create_slot;
+ walrcv_get_backend_pid_fn walrcv_get_backend_pid;
walrcv_exec_fn walrcv_exec;
walrcv_disconnect_fn walrcv_disconnect;
} WalReceiverFunctionsType;
@@ -276,6 +278,8 @@ extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
WalReceiverFunctions->walrcv_send(conn, buffer, nbytes)
#define walrcv_create_slot(conn, slotname, temporary, snapshot_action, lsn) \
WalReceiverFunctions->walrcv_create_slot(conn, slotname, temporary, snapshot_action, lsn)
+#define walrcv_get_backend_pid(conn) \+ WalReceiverFunctions->walrcv_get_backend_pid(conn)
#define walrcv_exec(conn, exec, nRetTypes, retTypes) \
WalReceiverFunctions->walrcv_exec(conn, exec, nRetTypes, retTypes)
#define walrcv_disconnect(conn) \
--
2.24.1
v2-0002-walreceiver-uses-a-temporary-replication-slot-by-.patch (text/plain, 8.2K) download
From c6ebd1275f8e2490c8a1a5dba981bdce53aafe20 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Sat, 11 Jan 2020 10:24:49 +0100
Subject: [PATCH v2 2/2] walreceiver uses a temporary replication slot by
default
If no permanent replication slot is configured using
primary_slot_name, the walreceiver now creates and uses a temporary
replication slot. A new setting wal_receiver_create_temp_slot can be
used to disable this behavior, for example, if the remote instance is
out of replication slots.
---
doc/src/sgml/config.sgml | 20 +++++++++
doc/src/sgml/monitoring.sgml | 9 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 4 ++
src/backend/replication/walreceiver.c | 41 +++++++++++++++++++
src/backend/utils/misc/guc.c | 9 ++++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/walreceiver.h | 7 ++++
7 files changed, 90 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgmlindex 5d1c90282f..5d45b6f7cb 100644--- a/doc/src/sgml/config.sgml+++ b/doc/src/sgml/config.sgml@@ -4124,6 +4124,26 @@ <title>Standby Servers</title>
</listitem>
</varlistentry>
+ <varlistentry id="guc-wal-receiver-create-temp-slot" xreflabel="wal_receiver_create_temp_slot">+ <term><varname>wal_receiver_create_temp_slot</varname> (<type>boolean</type>)+ <indexterm>+ <primary><varname>wal_receiver_create_temp_slot</varname> configuration parameter</primary>+ </indexterm>+ </term>+ <listitem>+ <para>+ Specifies whether a WAL receiver should create a temporary replication+ slot on the remote instance when no permanent replication slot to use+ has been configured (using <xref linkend="guc-primary-slot-name"/>).+ The default is on. The only reason to turn this off would be if the+ remote instance is currently out of available replication slots. This+ parameter can only be set in the <filename>postgresql.conf</filename>+ file or on the server command line. Changes only take effect when the+ WAL receiver process starts a new connection.+ </para>+ </listitem>+ </varlistentry>+
<varlistentry id="guc-wal-receiver-status-interval" xreflabel="wal_receiver_status_interval">
<term><varname>wal_receiver_status_interval</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgmlindex dcb58115af..a2f5bbae66 100644--- a/doc/src/sgml/monitoring.sgml+++ b/doc/src/sgml/monitoring.sgml@@ -2117,7 +2117,14 @@ <title><structname>pg_stat_wal_receiver</structname> View</title>
<row>
<entry><structfield>slot_name</structfield></entry>
<entry><type>text</type></entry>
- <entry>Replication slot name used by this WAL receiver</entry>+ <entry>+ Replication slot name used by this WAL receiver. This is only set if a+ permanent replication slot is set using <xref+ linkend="guc-primary-slot-name"/>. Otherwise, the WAL receiver may use+ a temporary replication slot (determined by <xref+ linkend="guc-wal-receiver-create-temp-slot"/>), but these are not shown+ here.+ </entry>
</row>
<row>
<entry><structfield>sender_host</structfield></entry>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.cindex b731d3fd04..e4fd1f9bb6 100644--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c@@ -834,6 +834,10 @@ libpqrcv_create_slot(WalReceiverConn *conn, const char *slotname,
break;
}
}
+ else+ {+ appendStringInfoString(&cmd, " PHYSICAL RESERVE_WAL");+ }
res = libpqrcv_PQexec(conn->streamConn, cmd.data);
pfree(cmd.data);
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.cindex 77360f1524..b464114333 100644--- a/src/backend/replication/walreceiver.c+++ b/src/backend/replication/walreceiver.c@@ -73,6 +73,7 @@
/* GUC variables */
+bool wal_receiver_create_temp_slot;
int wal_receiver_status_interval;
int wal_receiver_timeout;
bool hot_standby_feedback;
@@ -169,6 +170,7 @@ WalReceiverMain(void)
char conninfo[MAXCONNINFO];
char *tmp_conninfo;
char slotname[NAMEDATALEN];
+ bool is_temp_slot;
XLogRecPtr startpoint;
TimeLineID startpointTLI;
TimeLineID primaryTLI;
@@ -230,6 +232,7 @@ WalReceiverMain(void)
walrcv->ready_to_display = false;
strlcpy(conninfo, (char *) walrcv->conninfo, MAXCONNINFO);
strlcpy(slotname, (char *) walrcv->slotname, NAMEDATALEN);
+ is_temp_slot = walrcv->is_temp_slot;
startpoint = walrcv->receiveStart;
startpointTLI = walrcv->receiveStartTLI;
@@ -345,6 +348,44 @@ WalReceiverMain(void)
*/
WalRcvFetchTimeLineHistoryFiles(startpointTLI, primaryTLI);
+ /*+ * Create temporary replication slot if no slot name is configured or+ * the slot from the previous run was temporary, unless+ * wal_receiver_create_temp_slot is disabled. We also need to handle+ * the case where the previous run used a temporary slot but+ * wal_receiver_create_temp_slot was changed in the meantime. In that+ * case, we delete the old slot name in shared memory. (This would+ * all be a bit easier if we just didn't copy the slot name into+ * shared memory, since we won't need it again later, but then we+ * can't see the slot name in the stats views.)+ */+ if (slotname[0] == '\0' || is_temp_slot)+ {+ bool changed = false;++ if (wal_receiver_create_temp_slot)+ {+ snprintf(slotname, sizeof(slotname),+ "pg_walreceiver_%d", walrcv_get_backend_pid(wrconn));++ walrcv_create_slot(wrconn, slotname, true, 0, NULL);+ changed = true;+ }+ else if (slotname[0] != '\0')+ {+ slotname[0] = '\0';+ changed = true;+ }++ if (changed)+ {+ SpinLockAcquire(&walrcv->mutex);+ strlcpy(walrcv->slotname, slotname, NAMEDATALEN);+ walrcv->is_temp_slot = wal_receiver_create_temp_slot;+ SpinLockRelease(&walrcv->mutex);+ }+ }+
/*
* Start streaming.
*
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.cindex 62285792ec..e5f8a1301f 100644--- a/src/backend/utils/misc/guc.c+++ b/src/backend/utils/misc/guc.c@@ -1969,6 +1969,15 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {+ {"wal_receiver_create_temp_slot", PGC_SIGHUP, REPLICATION_STANDBY,+ gettext_noop("Sets whether a WAL receiver should create a temporary replication slot if no permanent slot is configured."),+ },+ &wal_receiver_create_temp_slot,+ true,+ NULL, NULL, NULL+ },+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, false, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sampleindex 087190ce63..e1048c0047 100644--- a/src/backend/utils/misc/postgresql.conf.sample+++ b/src/backend/utils/misc/postgresql.conf.sample@@ -321,6 +321,7 @@
#max_standby_streaming_delay = 30s # max delay before canceling queries
# when reading streaming WAL;
# -1 allows indefinite delay
+#wal_receiver_create_temp_slot = on # create temp slot if primary_slot_name not set
#wal_receiver_status_interval = 10s # send replies at least this often
# 0 disables
#hot_standby_feedback = off # send info from standby to prevent
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.hindex 172cfa2862..e08afc6548 100644--- a/src/include/replication/walreceiver.h+++ b/src/include/replication/walreceiver.h@@ -23,6 +23,7 @@
#include "utils/tuplestore.h"
/* user-settable parameters */
+extern bool wal_receiver_create_temp_slot;
extern int wal_receiver_status_interval;
extern int wal_receiver_timeout;
extern bool hot_standby_feedback;
@@ -121,6 +122,12 @@ typedef struct
*/
char slotname[NAMEDATALEN];
+ /*+ * If it's a temporary replication slot, it needs to be recreated when+ * connecting.+ */+ bool is_temp_slot;+
/* set true once conninfo is ready to display (obfuscated pwds etc) */
bool ready_to_display;
--
2.24.1
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: peter.eisentraut@2ndquadrant.com, masahiko.sawada@2ndquadrant.com, sk@zsrv.org, michael@paquier.xyz
Subject: Re: base backup client as auxiliary backend process
In-Reply-To: <4c79f835-b443-7b02-9097-aa1a40f46be9@2ndquadrant.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox