From: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
To: PostgreSQL-development <pgsql-hackers@postgresql.org>
Cc: Sergei Kornilov <sk@zsrv.org>
Cc: Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: Álvaro Herrera <alvherre@2ndquadrant.com>
Subject: Re: Improve handling of parameter differences in physical replication
Date: Wed, 24 Jun 2020 10:00:00 +0200
Message-ID: <ddd72267-d4cb-4b4e-07af-4685316a2c92@2ndquadrant.com> (raw)
In-Reply-To: <f71030f1-cd0a-5e7c-5d09-cbebc6603bf5@2ndquadrant.com>
References: <CA+fd4k4G8pFDha522kPAh_H2d3be6ejc56H932h_xHXPA+mV5Q@mail.gmail.com>
<20200310.175755.1250389993502025042.horikyota.ntt@gmail.com>
<aa078efe-2d25-fdab-5bdb-158edab5d849@2ndquadrant.com>
<20200311.110637.1319131984113864409.horikyota.ntt@gmail.com>
<2134a29f-e934-4585-e81f-ab416e1923be@2ndquadrant.com>
<CA+fd4k4G3pAY+KPDzNRCGu=FZQT81KhXT-8EcjYGpc6cpmiT9w@mail.gmail.com>
<2808491585335240@iva3-67f911cb3a01.qloud-c.yandex.net>
<f71030f1-cd0a-5e7c-5d09-cbebc6603bf5@2ndquadrant.com>
Here is another stab at this subject.
This is a much simplified variant: When encountering a parameter change
in the WAL that is higher than the standby's current setting, we log a
warning (instead of an error until now) and pause recovery. If you
resume (unpause) recovery, the instance shuts down as before.
This allows you to keep your standbys running for a bit (depending on
lag requirements) and schedule the required restart more deliberately.
I had previously suggested making this new behavior configurable, but
there didn't seem to be much interest in that, so I have not included
that there.
The documentation changes are mostly carried over from previous patch
versions (but adjusted for the actual behavior of the patch).
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
From 2d3ffb221ae9418ebd7e1fc7fb771213014ca216 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Wed, 24 Jun 2020 08:49:42 +0200
Subject: [PATCH v3 1/2] Replace a macro by a function
Using a macro is ugly and not justified here.
---
src/backend/access/transam/xlog.c | 21 +++++++++++----------
1 file changed, 11 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.cindex a1256a103b..dcec2111b3 100644--- a/src/backend/access/transam/xlog.c+++ b/src/backend/access/transam/xlog.c@@ -6232,16 +6232,17 @@ GetXLogReceiptTime(TimestampTz *rtime, bool *fromStream)
* Note that text field supplied is a parameter name and does not require
* translation
*/
-#define RecoveryRequiresIntParameter(param_name, currValue, minValue) \-do { \- if ((currValue) < (minValue)) \- ereport(ERROR, \- (errcode(ERRCODE_INVALID_PARAMETER_VALUE), \- errmsg("hot standby is not possible because %s = %d is a lower setting than on the master server (its value was %d)", \- param_name, \- currValue, \- minValue))); \-} while(0)+static void+RecoveryRequiresIntParameter(const char *param_name, int currValue, int minValue)+{+ if (currValue < minValue)+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("hot standby is not possible because %s = %d is a lower setting than on the master server (its value was %d)",+ param_name,+ currValue,+ minValue)));+}
/*
* Check to see if required parameters are set high enough on this server
--
2.27.0
From 18740a021a90207423032cb7201f54650b81c15a Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Wed, 24 Jun 2020 09:18:21 +0200
Subject: [PATCH v3 2/2] Pause recovery for insufficient parameter settings
When certain parameters are changed on a physical replication primary,
this is communicated to standbys using the XLOG_PARAMETER_CHANGE WAL
record. The standby then checks whether its own settings are at least
as big as the ones on the primary. If not, the standby shuts down
with a fatal error.
This patch changes this behavior to pause recovery at that point
instead. That allows read traffic on the standby to continue while
database administrators figure out next steps. When recovery is
unpaused, the server shuts down (as before). The idea is to fix the
parameters while recovery is paused and then restart when there is a
maintenance window.
---
doc/src/sgml/high-availability.sgml | 47 +++++++++++++++++++++--------
src/backend/access/transam/xlog.c | 17 ++++++++++-
2 files changed, 51 insertions(+), 13 deletions(-)
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgmlindex 65c3fc62a9..d32ec6d9b3 100644--- a/doc/src/sgml/high-availability.sgml+++ b/doc/src/sgml/high-availability.sgml@@ -2159,18 +2159,14 @@ <title>Administrator's Overview</title>
</para>
<para>
- The setting of some parameters on the standby will need reconfiguration- if they have been changed on the primary. For these parameters,- the value on the standby must- be equal to or greater than the value on the primary.- Therefore, if you want to increase these values, you should do so on all- standby servers first, before applying the changes to the primary server.- Conversely, if you want to decrease these values, you should do so on the- primary server first, before applying the changes to all standby servers.- If these parameters- are not set high enough then the standby will refuse to start.- Higher values can then be supplied and the server- restarted to begin recovery again. These parameters are:+ The settings of some parameters determine the size of shared memory for+ tracking transaction IDs, locks, and prepared transactions. These shared+ memory structures must be no smaller on a standby than on the primary in+ order to ensure that the standby does not run out of shared memory during+ recovery. For example, if the primary had used a prepared transaction but+ the standby had not allocated any shared memory for tracking prepared+ transactions, then recovery could not continue until the standby's+ configuration is changed. The parameters affected are:
<itemizedlist>
<listitem>
@@ -2199,6 +2195,33 @@ <title>Administrator's Overview</title>
</para>
</listitem>
</itemizedlist>
++ The easiest way to ensure this does not become a problem is to have these+ parameters set on the standbys to values equal to or greater than on the+ primary. Therefore, if you want to increase these values, you should do+ so on all standby servers first, before applying the changes to the+ primary server. Conversely, if you want to decrease these values, you+ should do so on the primary server first, before applying the changes to+ all standby servers. Keep in mind that when a standby is promoted, it+ becomes the new reference for the required parameter settings for the+ standbys that follow it. Therefore, to avoid this becoming a problem+ during a switchover or failover, it is recommended to keep these settings+ the same on all standby servers.+ </para>++ <para>+ The WAL tracks changes to these parameters on the+ primary, and if a standby processes WAL that indicates that the current+ value on the primary is higher than its own value, it will log a warning+ and pause recovery, for example:+<screen>+WARNING: hot standby is not possible because max_connections = 80 is a lower setting than on the master server (its value was 100)+LOG: recovery will be paused+DETAIL: If recovery is unpaused, the server will shut down.+HINT: You can then restart the server after making the necessary configuration changes.+</screen>+ At that point, the settings on the standby need to be updated and the+ instance restarted before recovery can continue.
</para>
<para>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.cindex dcec2111b3..e0e22e26f4 100644--- a/src/backend/access/transam/xlog.c+++ b/src/backend/access/transam/xlog.c@@ -6236,12 +6236,27 @@ static void
RecoveryRequiresIntParameter(const char *param_name, int currValue, int minValue)
{
if (currValue < minValue)
- ereport(ERROR,+ {+ ereport(WARNING,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("hot standby is not possible because %s = %d is a lower setting than on the master server (its value was %d)",
param_name,
currValue,
minValue)));
+ ereport(LOG,+ (errmsg("recovery will be paused"),+ errdetail("If recovery is unpaused, the server will shut down."),+ errhint("You can then restart the server after making the necessary configuration changes.")));+ SetRecoveryPause(true);+ while (RecoveryIsPaused())+ {+ pg_usleep(1000000L); /* 1000 ms */+ HandleStartupProcInterrupts();+ }+ ereport(FATAL,+ (errmsg("recovery aborted because of insufficient parameter settings"),+ errhint("You can restart the server after making the necessary configuration changes.")));+ }
}
/*
--
2.27.0
Attachments:
[text/plain] v3-0001-Replace-a-macro-by-a-function.patch (1.6K, ../ddd72267-d4cb-4b4e-07af-4685316a2c92@2ndquadrant.com/2-v3-0001-Replace-a-macro-by-a-function.patch)
download | inline diff:
From 2d3ffb221ae9418ebd7e1fc7fb771213014ca216 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Wed, 24 Jun 2020 08:49:42 +0200
Subject: [PATCH v3 1/2] Replace a macro by a function
Using a macro is ugly and not justified here.
---
src/backend/access/transam/xlog.c | 21 +++++++++++----------
1 file changed, 11 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.cindex a1256a103b..dcec2111b3 100644--- a/src/backend/access/transam/xlog.c+++ b/src/backend/access/transam/xlog.c@@ -6232,16 +6232,17 @@ GetXLogReceiptTime(TimestampTz *rtime, bool *fromStream)
* Note that text field supplied is a parameter name and does not require
* translation
*/
-#define RecoveryRequiresIntParameter(param_name, currValue, minValue) \-do { \- if ((currValue) < (minValue)) \- ereport(ERROR, \- (errcode(ERRCODE_INVALID_PARAMETER_VALUE), \- errmsg("hot standby is not possible because %s = %d is a lower setting than on the master server (its value was %d)", \- param_name, \- currValue, \- minValue))); \-} while(0)+static void+RecoveryRequiresIntParameter(const char *param_name, int currValue, int minValue)+{+ if (currValue < minValue)+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("hot standby is not possible because %s = %d is a lower setting than on the master server (its value was %d)",+ param_name,+ currValue,+ minValue)));+}
/*
* Check to see if required parameters are set high enough on this server
--
2.27.0
[text/plain] v3-0002-Pause-recovery-for-insufficient-parameter-setting.patch (5.5K, ../ddd72267-d4cb-4b4e-07af-4685316a2c92@2ndquadrant.com/3-v3-0002-Pause-recovery-for-insufficient-parameter-setting.patch)
download | inline diff:
From 18740a021a90207423032cb7201f54650b81c15a Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Wed, 24 Jun 2020 09:18:21 +0200
Subject: [PATCH v3 2/2] Pause recovery for insufficient parameter settings
When certain parameters are changed on a physical replication primary,
this is communicated to standbys using the XLOG_PARAMETER_CHANGE WAL
record. The standby then checks whether its own settings are at least
as big as the ones on the primary. If not, the standby shuts down
with a fatal error.
This patch changes this behavior to pause recovery at that point
instead. That allows read traffic on the standby to continue while
database administrators figure out next steps. When recovery is
unpaused, the server shuts down (as before). The idea is to fix the
parameters while recovery is paused and then restart when there is a
maintenance window.
---
doc/src/sgml/high-availability.sgml | 47 +++++++++++++++++++++--------
src/backend/access/transam/xlog.c | 17 ++++++++++-
2 files changed, 51 insertions(+), 13 deletions(-)
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgmlindex 65c3fc62a9..d32ec6d9b3 100644--- a/doc/src/sgml/high-availability.sgml+++ b/doc/src/sgml/high-availability.sgml@@ -2159,18 +2159,14 @@ <title>Administrator's Overview</title>
</para>
<para>
- The setting of some parameters on the standby will need reconfiguration- if they have been changed on the primary. For these parameters,- the value on the standby must- be equal to or greater than the value on the primary.- Therefore, if you want to increase these values, you should do so on all- standby servers first, before applying the changes to the primary server.- Conversely, if you want to decrease these values, you should do so on the- primary server first, before applying the changes to all standby servers.- If these parameters- are not set high enough then the standby will refuse to start.- Higher values can then be supplied and the server- restarted to begin recovery again. These parameters are:+ The settings of some parameters determine the size of shared memory for+ tracking transaction IDs, locks, and prepared transactions. These shared+ memory structures must be no smaller on a standby than on the primary in+ order to ensure that the standby does not run out of shared memory during+ recovery. For example, if the primary had used a prepared transaction but+ the standby had not allocated any shared memory for tracking prepared+ transactions, then recovery could not continue until the standby's+ configuration is changed. The parameters affected are:
<itemizedlist>
<listitem>
@@ -2199,6 +2195,33 @@ <title>Administrator's Overview</title>
</para>
</listitem>
</itemizedlist>
++ The easiest way to ensure this does not become a problem is to have these+ parameters set on the standbys to values equal to or greater than on the+ primary. Therefore, if you want to increase these values, you should do+ so on all standby servers first, before applying the changes to the+ primary server. Conversely, if you want to decrease these values, you+ should do so on the primary server first, before applying the changes to+ all standby servers. Keep in mind that when a standby is promoted, it+ becomes the new reference for the required parameter settings for the+ standbys that follow it. Therefore, to avoid this becoming a problem+ during a switchover or failover, it is recommended to keep these settings+ the same on all standby servers.+ </para>++ <para>+ The WAL tracks changes to these parameters on the+ primary, and if a standby processes WAL that indicates that the current+ value on the primary is higher than its own value, it will log a warning+ and pause recovery, for example:+<screen>+WARNING: hot standby is not possible because max_connections = 80 is a lower setting than on the master server (its value was 100)+LOG: recovery will be paused+DETAIL: If recovery is unpaused, the server will shut down.+HINT: You can then restart the server after making the necessary configuration changes.+</screen>+ At that point, the settings on the standby need to be updated and the+ instance restarted before recovery can continue.
</para>
<para>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.cindex dcec2111b3..e0e22e26f4 100644--- a/src/backend/access/transam/xlog.c+++ b/src/backend/access/transam/xlog.c@@ -6236,12 +6236,27 @@ static void
RecoveryRequiresIntParameter(const char *param_name, int currValue, int minValue)
{
if (currValue < minValue)
- ereport(ERROR,+ {+ ereport(WARNING,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("hot standby is not possible because %s = %d is a lower setting than on the master server (its value was %d)",
param_name,
currValue,
minValue)));
+ ereport(LOG,+ (errmsg("recovery will be paused"),+ errdetail("If recovery is unpaused, the server will shut down."),+ errhint("You can then restart the server after making the necessary configuration changes.")));+ SetRecoveryPause(true);+ while (RecoveryIsPaused())+ {+ pg_usleep(1000000L); /* 1000 ms */+ HandleStartupProcInterrupts();+ }+ ereport(FATAL,+ (errmsg("recovery aborted because of insufficient parameter settings"),+ errhint("You can restart the server after making the necessary configuration changes.")));+ }
}
/*
--
2.27.0
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: peter.eisentraut@2ndquadrant.com, sk@zsrv.org, masahiko.sawada@2ndquadrant.com, horikyota.ntt@gmail.com, alvherre@2ndquadrant.com
Subject: Re: Improve handling of parameter differences in physical replication
In-Reply-To: <ddd72267-d4cb-4b4e-07af-4685316a2c92@2ndquadrant.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox