public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 5/6] Documentation for slot-limit feature
68+ messages / 10 participants
[nested] [flat]
* [PATCH 5/6] Documentation for slot-limit feature
@ 2018-01-11 06:00 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 68+ messages in thread
From: Kyotaro Horiguchi @ 2018-01-11 06:00 UTC (permalink / raw)
---
doc/src/sgml/catalogs.sgml | 30 ++++++++++++++++++++++++++++++
doc/src/sgml/config.sgml | 23 +++++++++++++++++++++++
doc/src/sgml/high-availability.sgml | 8 +++++---
3 files changed, 58 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 68ad5071ca..dc9679283a 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -9975,6 +9975,36 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</entry>
</row>
+ <row>
+ <entry><structfield>wal_status</structfield></entry>
+ <entry><type>text</type></entry>
+ <entry></entry>
+
+ <entry>Availability of WAL records claimed by this
+ slot. <literal>streaming</literal>, <literal>keeping</literal>,
+ <literal>lost</literal>
+ or <literal>unknown</literal>. <literal>streaming</literal> means that
+ the claimed records are available within
+ max_wal_size. <literal>keeping</literal> means max_wal_size is exceeded
+ but still required records are held by replication slots or
+ wal_keep_segments.
+ <literal>lost</literal> means that some of them are on the verge of
+ removal or no longer available. This state is seen only when
+ <xref linkend="guc-max-slot-wal-keep-size"/> is
+ non-negative. If <structfield>restart_lsn</structfield> is NULL, this
+ field is <literal>unknown</literal>.
+ </entry>
+ </row>
+
+ <row>
+ <entry><structfield>remain</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry></entry>
+ <entry>The amount in bytes that WAL location (LSN) can advance until
+ this slot may lose required WAL records.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index c91e3e1550..c345538c8f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -3650,6 +3650,29 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-max-slot-wal-keep-size" xreflabel="max_slot_wal_keep_size">
+ <term><varname>max_slot_wal_keep_size</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>max_slot_wal_keep_size</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specify the maximum size of WAL files
+ that <link linkend="streaming-replication-slots">replication
+ slots</link> are allowed to retain in the <filename>pg_wal</filename>
+ directory at checkpoint time.
+ If <varname>max_slot_wal_keep_size</varname> is -1 (the default),
+ replication slots retain unlimited amount of WAL files. If
+ restart_lsn of a replication slot gets behind more than that megabytes
+ from the current LSN, the standby using the slot may no longer be able
+ to continue replication due to removal of required WAL records. You
+ can see the WAL availability of replication slots
+ in <link linkend="view-pg-replication-slots">pg_replication_slots</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-wal-sender-timeout" xreflabel="wal_sender_timeout">
<term><varname>wal_sender_timeout</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 543691dad4..ae8c3a2aca 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -925,9 +925,11 @@ primary_conninfo = 'host=192.168.1.50 port=5432 user=foo password=foopass'
<xref linkend="guc-archive-command"/>.
However, these methods often result in retaining more WAL segments than
required, whereas replication slots retain only the number of segments
- known to be needed. An advantage of these methods is that they bound
- the space requirement for <literal>pg_wal</literal>; there is currently no way
- to do this using replication slots.
+ known to be needed. On the other hand, replication slots can retain so
+ many WAL segments that they fill up the space allocated
+ for <literal>pg_wal</literal>;
+ <xref linkend="guc-max-slot-wal-keep-size"/> limits the size of WAL files
+ retained by replication slots.
</para>
<para>
Similarly, <xref linkend="guc-hot-standby-feedback"/>
--
2.16.3
----Next_Part(Tue_Jul_30_21_30_45_2019_680)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v15-0006-Check-removal-of-in-reading-segment-file.patch"
^ permalink raw reply [nested|flat] 68+ messages in thread
* [PATCH v29 05/11] Add Incremental View Maintenance support to psql
@ 2019-12-20 01:21 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 68+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:21 UTC (permalink / raw)
Add tab completion and meta-command output for IVM.
---
src/bin/psql/describe.c | 32 +++++++++++++++++++++++++++++++-
src/bin/psql/tab-complete.c | 14 +++++++++-----
2 files changed, 40 insertions(+), 6 deletions(-)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index bac94a338c..f6c7e7163d 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -1575,6 +1575,7 @@ describeOneTableDetails(const char *schemaname,
char relpersistence;
char relreplident;
char *relam;
+ bool isivm;
} tableinfo;
bool show_column_details = false;
@@ -1587,7 +1588,26 @@ describeOneTableDetails(const char *schemaname,
initPQExpBuffer(&tmpbuf);
/* Get general table info */
- if (pset.sversion >= 120000)
+ if (pset.sversion >= 170000)
+ {
+ printfPQExpBuffer(&buf,
+ "SELECT c.relchecks, c.relkind, c.relhasindex, c.relhasrules, "
+ "c.relhastriggers, c.relrowsecurity, c.relforcerowsecurity, "
+ "false AS relhasoids, c.relispartition, %s, c.reltablespace, "
+ "CASE WHEN c.reloftype = 0 THEN '' ELSE c.reloftype::pg_catalog.regtype::pg_catalog.text END, "
+ "c.relpersistence, c.relreplident, am.amname, "
+ "c.relisivm\n"
+ "FROM pg_catalog.pg_class c\n "
+ "LEFT JOIN pg_catalog.pg_class tc ON (c.reltoastrelid = tc.oid)\n"
+ "LEFT JOIN pg_catalog.pg_am am ON (c.relam = am.oid)\n"
+ "WHERE c.oid = '%s';",
+ (verbose ?
+ "pg_catalog.array_to_string(c.reloptions || "
+ "array(select 'toast.' || x from pg_catalog.unnest(tc.reloptions) x), ', ')\n"
+ : "''"),
+ oid);
+ }
+ else if (pset.sversion >= 120000)
{
printfPQExpBuffer(&buf,
"SELECT c.relchecks, c.relkind, c.relhasindex, c.relhasrules, "
@@ -1707,6 +1727,10 @@ describeOneTableDetails(const char *schemaname,
(char *) NULL : pg_strdup(PQgetvalue(res, 0, 14));
else
tableinfo.relam = NULL;
+ if (pset.sversion >= 170000)
+ tableinfo.isivm = strcmp(PQgetvalue(res, 0, 15), "t") == 0;
+ else
+ tableinfo.isivm = false;
PQclear(res);
res = NULL;
@@ -3552,6 +3576,12 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, _("Access method: %s"), tableinfo.relam);
printTableAddFooter(&cont, buf.data);
}
+
+ /* Incremental view maintance info */
+ if (verbose && tableinfo.relkind == RELKIND_MATVIEW && tableinfo.isivm)
+ {
+ printTableAddFooter(&cont, _("Incremental view maintenance: yes"));
+ }
}
/* reloptions, if verbose */
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 779fdc90cb..9cc79b722f 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1244,6 +1244,7 @@ static const pgsql_thing_t words_after_create[] = {
{"FOREIGN TABLE", NULL, NULL, NULL},
{"FUNCTION", NULL, NULL, Query_for_list_of_functions},
{"GROUP", Query_for_list_of_roles},
+ {"INCREMENTAL MATERIALIZED VIEW", NULL, NULL, &Query_for_list_of_matviews, NULL, THING_NO_DROP | THING_NO_ALTER},
{"INDEX", NULL, NULL, &Query_for_list_of_indexes},
{"LANGUAGE", Query_for_list_of_languages},
{"LARGE OBJECT", NULL, NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
@@ -3217,7 +3218,7 @@ psql_completion(const char *text, int start, int end)
if (HeadMatches("CREATE", "SCHEMA"))
COMPLETE_WITH("TABLE", "SEQUENCE");
else
- COMPLETE_WITH("TABLE", "SEQUENCE", "MATERIALIZED VIEW");
+ COMPLETE_WITH("TABLE", "SEQUENCE", "MATERIALIZED VIEW", "INCREMENTAL MATERIALIZED VIEW");
}
/* Complete PARTITION BY with RANGE ( or LIST ( or ... */
else if (TailMatches("PARTITION", "BY"))
@@ -3535,13 +3536,16 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("SELECT");
/* CREATE MATERIALIZED VIEW */
- else if (Matches("CREATE", "MATERIALIZED"))
+ else if (Matches("CREATE", "MATERIALIZED") ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED"))
COMPLETE_WITH("VIEW");
- /* Complete CREATE MATERIALIZED VIEW <name> with AS */
- else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny))
+ /* Complete CREATE MATERIALIZED VIEW <name> with AS */
+ else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny) ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny))
COMPLETE_WITH("AS");
/* Complete "CREATE MATERIALIZED VIEW <sth> AS with "SELECT" */
- else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS"))
+ else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS") ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny, "AS"))
COMPLETE_WITH("SELECT");
/* CREATE EVENT TRIGGER */
--
2.25.1
--Multipart=_Mon__28_Aug_2023_11_52_52_+0900_hj6L5h176QaSGtg7
Content-Type: text/x-diff;
name="v29-0006-Add-Incremental-View-Maintenance-support.patch"
Content-Disposition: attachment;
filename="v29-0006-Add-Incremental-View-Maintenance-support.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-02-27 01:42 Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-02-27 01:42 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Monday, February 26, 2024 1:19 PM Amit Kapila <[email protected]> wrote:
>
> On Fri, Feb 23, 2024 at 4:45 PM Bertrand Drouvot
> <[email protected]> wrote:
> >
> > On Fri, Feb 23, 2024 at 09:46:00AM +0000, Zhijie Hou (Fujitsu) wrote:
> > >
> > > Besides, I'd like to clarify and discuss the behavior of standby_slot_names
> once.
> > >
> > > As it stands in the patch, If the slots specified in
> > > standby_slot_names are dropped or invalidated, the logical walsender
> > > will issue a WARNING and continue to replicate the changes. Another
> > > option for this could be to have the walsender pause until the slot
> > > in standby_slot_names is re-created or becomes valid again. Does anyone
> else have an opinion on this matter ?
> >
> > Good point, I'd vote for: the only reasons not to wait are:
> >
> > - slots mentioned in standby_slot_names exist and valid and do catch
> > up or
> > - standby_slot_names is empty
> >
> > The reason is that setting standby_slot_names to a non empty value
> > means that one wants the walsender to wait until the standby catchup.
> > The way to remove this intentional behavior should be by changing the
> > standby_slot_names value (not the existence or the state of the slot(s) it
> points too).
> >
>
> It seems we already do wait for the case when there is an inactive slot as per the
> below code [1] in the patch. So, probably waiting in other cases is also okay and
> also as this parameter is a SIGHUP parameter, users should be easily able to
> change its value if required. Do you think it is a good idea to mention this in
> docs as well?
>
> I think it is important to raise WARNING as the patch is doing in all the cases
> where the slot is not being processed so that users can be notified and they can
> take the required action.
Agreed. Here is the V99 patch which addressed the above.
This version also includes:
1. list_free the slot list when reloading the list due to GUC change.
2. Refactored the validate_standby_slots based on Shveta's suggestion.
3. Added errcode for the warnings as most of existing have errcodes.
Amit's latest comments[1] are pending, we will address that in next version.
[1] https://www.postgresql.org/message-id/CAA4eK1LJdmGATWG%3DxOD1CB9cogukk2cLNBGH8h-n-ZDJuwBdJg%40mail.g...
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v99-0002-Document-the-steps-to-check-if-the-standby-is-re.patch (7.0K, ../../OS0PR01MB571654A0288C53A676051AE794592@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v99-0002-Document-the-steps-to-check-if-the-standby-is-re.patch)
download | inline diff:
From 52b3df7d98c3447223d26c3860ff921f19fb6e27 Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Fri, 19 Jan 2024 11:04:16 +0530
Subject: [PATCH v99 2/2] Document the steps to check if the standby is ready
for failover
---
doc/src/sgml/high-availability.sgml | 9 ++
doc/src/sgml/logical-replication.sgml | 136 ++++++++++++++++++++++++++
2 files changed, 145 insertions(+)
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 236c0af65f..36215aa68c 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -1487,6 +1487,15 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)'
Written administration procedures are advised.
</para>
+ <para>
+ If you have opted for synchronization of logical slots (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>),
+ then before switching to the standby server, it is recommended to check
+ if the logical slots synchronized on the standby server are ready
+ for failover. This can be done by following the steps described in
+ <xref linkend="logical-replication-failover"/>.
+ </para>
+
<para>
To trigger failover of a log-shipping standby server, run
<command>pg_ctl promote</command> or call <function>pg_promote()</function>.
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ec2130669e..be59d306a1 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -687,6 +687,142 @@ ALTER SUBSCRIPTION
</sect1>
+ <sect1 id="logical-replication-failover">
+ <title>Logical Replication Failover</title>
+
+ <para>
+ When the publisher server is the primary server of a streaming replication,
+ the logical slots on that primary server can be synchronized to the standby
+ server by specifying <literal>failover = true</literal> when creating
+ subscriptions for those publications. Enabling failover ensures a seamless
+ transition of those subscriptions after the standby is promoted. They can
+ continue subscribing to publications now on the new primary server without
+ any data loss.
+ </para>
+
+ <para>
+ Because the slot synchronization logic copies asynchronously, it is
+ necessary to confirm that replication slots have been synced to the standby
+ server before the failover happens. Furthermore, to ensure a successful
+ failover, the standby server must not be lagging behind the subscriber. It
+ is highly recommended to use
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ to prevent the subscriber from consuming changes faster than the hot standby.
+ To confirm that the standby server is indeed ready for failover, follow
+ these 2 steps:
+ </para>
+
+ <procedure>
+ <step performance="required">
+ <para>
+ Confirm that all the necessary logical replication slots have been synced to
+ the standby server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node, use the following SQL to identify
+ which slots should be synced to the standby that we plan to promote.
+<programlisting>
+test_sub=# SELECT
+ array_agg(slotname) AS slots
+ FROM
+ ((
+ SELECT r.srsubid AS subid, CONCAT('pg_', srsubid, '_sync_', srrelid, '_', ctl.system_identifier) AS slotname
+ FROM pg_control_system() ctl, pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate = 'f' AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT s.oid AS subid, s.subslotname as slotname
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ slots
+-------
+ {sub1,sub2,sub3}
+(1 row)
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, check that the logical replication slots identified above exist on
+ the standby server and are ready for failover.
+<programlisting>
+test_standby=# SELECT slot_name, (synced AND NOT temporary AND conflict_reason IS NULL) AS failover_ready
+ FROM pg_replication_slots
+ WHERE slot_name IN ('sub1','sub2','sub3');
+ slot_name | failover_ready
+-------------+----------------
+ sub1 | t
+ sub2 | t
+ sub3 | t
+(3 rows)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+
+ <step performance="required">
+ <para>
+ Confirm that the standby server is not lagging behind the subscribers.
+ This step can be skipped if
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ has been correctly configured. If standby_slot_names is not configured
+ correctly, it is highly recommended to run this step after the primary
+ server is down, otherwise the results of the query may vary at different
+ points of time due to the ongoing replication on the logical subscribers
+ from the primary server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node check the last replayed WAL.
+ This step needs to be run on the database(s) that includes the failover
+ enabled subscription(s), to find the last replayed WAL on each database.
+<programlisting>
+test_sub=# SELECT
+ MAX(remote_lsn) AS remote_lsn_on_subscriber
+ FROM
+ ((
+ SELECT (CASE WHEN r.srsubstate = 'f' THEN pg_replication_origin_progress(CONCAT('pg_', r.srsubid, '_', r.srrelid), false)
+ WHEN r.srsubstate IN ('s', 'r') THEN r.srsublsn END) AS remote_lsn
+ FROM pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate IN ('f', 's', 'r') AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT pg_replication_origin_progress(CONCAT('pg_', s.oid), false) AS remote_lsn
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ remote_lsn_on_subscriber
+--------------------------
+ 0/3000388
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, on the standby server check that the last-received WAL location
+ is ahead of the replayed WAL location(s) on the subscriber identified
+ above. If the above SQL result was NULL, it means the subscriber has not
+ yet replayed any WAL, so the standby server must be ahead of the
+ subscriber, and this step can be skipped.
+<programlisting>
+test_standby=# SELECT pg_last_wal_receive_lsn() >= '0/3000388'::pg_lsn AS failover_ready;
+ failover_ready
+----------------
+ t
+(1 row)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+ </procedure>
+
+ <para>
+ If the result (<literal>failover_ready</literal>) of both above steps is
+ true, existing subscriptions will be able to continue without data loss.
+ </para>
+
+ </sect1>
+
<sect1 id="logical-replication-row-filter">
<title>Row Filters</title>
--
2.30.0.windows.2
[application/octet-stream] v99-0001-Allow-logical-walsenders-to-wait-for-the-physica.patch (37.1K, ../../OS0PR01MB571654A0288C53A676051AE794592@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v99-0001-Allow-logical-walsenders-to-wait-for-the-physica.patch)
download | inline diff:
From 0e19341d364701150a1f914b22006a1f103b621e Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Fri, 23 Feb 2024 08:29:30 +0800
Subject: [PATCH v99 1/2] Allow logical walsenders to wait for the physical
standby
This patch introduces a mechanism to ensure that physical standby servers,
which are potential failover candidates, have received and flushed changes
before making them visible to subscribers. By doing so, it guarantees that
the promoted standby server is not lagging behind the subscribers when a
failover is necessary.
A new parameter named standby_slot_names is introduced. The logical
walsender now guarantees that all local changes are sent and flushed to
the standby servers corresponding to the replication slots specified in
standby_slot_names before sending those changes to the subscriber.
Additionally, The SQL functions pg_logical_slot_get_changes and
pg_replication_slot_advance are modified to wait for the replication slots
mentioned in standby_slot_names to catch up before returning the changes
to the user.
---
doc/src/sgml/config.sgml | 25 ++
doc/src/sgml/logicaldecoding.sgml | 8 +
.../replication/logical/logicalfuncs.c | 12 +
src/backend/replication/logical/slotsync.c | 13 +
src/backend/replication/slot.c | 300 +++++++++++++++++-
src/backend/replication/slotfuncs.c | 8 +
src/backend/replication/walsender.c | 114 ++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 14 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/replication/slot.h | 6 +
src/include/replication/walsender.h | 1 +
src/include/replication/walsender_private.h | 7 +
src/include/utils/guc_hooks.h | 3 +
src/test/recovery/t/006_logical_decoding.pl | 3 +-
.../t/040_standby_failover_slots_sync.pl | 184 +++++++++++
16 files changed, 685 insertions(+), 16 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 36a2a5ce43..d1e0a3bd2f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4420,6 +4420,31 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ List of physical slots guarantees that logical replication slots with
+ failover enabled do not consume changes until those changes are received
+ and flushed to corresponding physical standbys. If a logical replication
+ connection is meant to switch to a physical standby after the standby is
+ promoted, the physical replication slot for the standby should be listed
+ here. Note that logical replication will not proceed if the slots
+ specified in the standby_slot_names do not exist or are invalidated.
+ </para>
+ <para>
+ The standbys corresponding to the physical replication slots in
+ <varname>standby_slot_names</varname> must configure
+ <literal>sync_replication_slots = true</literal> so they can receive
+ failover logical slots changes from the primary.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 930c0fa8a6..73b2abeda5 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -384,6 +384,14 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU
must be enabled on the standby. It is also necessary to specify a valid
<literal>dbname</literal> in the
<link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>.
+ It's also highly recommended that the said physical replication slot
+ is named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ list on the primary, to prevent the subscriber from consuming changes
+ faster than the hot standby. But once we configure it, then certain latency
+ is expected in sending changes to logical subscribers due to wait on
+ physical replication slots in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
</para>
<para>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b0081d3ce5..0cd09c0e1c 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -109,6 +109,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
MemoryContext per_query_ctx;
MemoryContext oldcontext;
XLogRecPtr end_of_wal;
+ XLogRecPtr wait_for_wal_lsn;
LogicalDecodingContext *ctx;
ResourceOwner old_resowner = CurrentResourceOwner;
ArrayType *arr;
@@ -228,6 +229,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
NameStr(MyReplicationSlot->data.plugin),
format_procedure(fcinfo->flinfo->fn_oid))));
+ if (XLogRecPtrIsInvalid(upto_lsn))
+ wait_for_wal_lsn = end_of_wal;
+ else
+ wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
+
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to wait_for_wal_lsn.
+ */
+ WaitForStandbyConfirmation(wait_for_wal_lsn);
+
ctx->output_writer_private = p;
/*
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 36773cfe73..de05389d61 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -491,6 +491,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
latestFlushPtr = GetStandbyFlushRecPtr(NULL);
if (remote_slot->confirmed_lsn > latestFlushPtr)
{
+ /*
+ * Can get here only if GUC 'standby_slot_names' on the primary server
+ * was not configured correctly.
+ */
ereport(am_slotsync_worker ? LOG : ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("skipping slot synchronization as the received slot sync"
@@ -860,6 +864,15 @@ validate_remote_info(WalReceiverConn *wrconn)
remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull));
Assert(!isnull);
+ /*
+ * Slot sync is currently not supported on a cascading standby. This is
+ * because if we allow it, the primary server needs to wait for all the
+ * cascading standbys, otherwise, logical subscribers can still be ahead
+ * of one of the cascading standbys which we plan to promote. Thus, to
+ * avoid this additional complexity, we restrict it for the time being.
+ *
+ * XXX: If needed, this can be attempted in future.
+ */
if (remote_in_recovery)
ereport(ERROR,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 0f173f63a2..d149939d3b 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -46,13 +46,18 @@
#include "common/string.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/interrupt.h"
#include "replication/slotsync.h"
#include "replication/slot.h"
+#include "replication/walsender_private.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
+#include "utils/memutils.h"
+#include "utils/varlena.h"
/*
* Replication slot on-disk data structure.
@@ -115,10 +120,19 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
/* My backend's replication slot in the shared memory array */
ReplicationSlot *MyReplicationSlot = NULL;
-/* GUC variable */
+/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+/*
+ * This GUC lists streaming replication standby server slot names that
+ * logical WAL sender processes will wait for.
+ */
+char *standby_slot_names;
+
+/* This is parsed and cached list for raw standby_slot_names. */
+static List *standby_slot_names_list = NIL;
+
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
@@ -2345,3 +2359,287 @@ GetSlotInvalidationCause(const char *conflict_reason)
Assert(found);
return result;
}
+
+/*
+ * A helper function to validate slots specified in GUC standby_slot_names.
+ */
+static bool
+validate_standby_slots(char **newval)
+{
+ char *rawname;
+ List *elemlist;
+ ListCell *lc;
+ bool ok;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Verify syntax and parse string into a list of identifiers */
+ ok = SplitIdentifierString(rawname, ',', &elemlist);
+
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
+
+ /*
+ * If the replication slots' data have been initialized, verify if the
+ * specified slots exist and are logical slots.
+ */
+ else if (ReplicationSlotCtl)
+ {
+ foreach(lc, elemlist)
+ {
+ char *name = lfirst(lc);
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, true);
+
+ if (!slot)
+ {
+ GUC_check_errdetail("replication slot \"%s\" does not exist",
+ name);
+ ok = false;
+ break;
+ }
+
+ if (!SlotIsPhysical(slot))
+ {
+ GUC_check_errdetail("\"%s\" is not a physical replication slot",
+ name);
+ ok = false;
+ break;
+ }
+ }
+ }
+
+ pfree(rawname);
+ list_free(elemlist);
+ return ok;
+}
+
+/*
+ * GUC check_hook for standby_slot_names
+ */
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ if (strcmp(*newval, "") == 0)
+ return true;
+
+ /*
+ * "*" is not accepted as in that case primary will not be able to know
+ * for which all standbys to wait for. Even if we have physical slots
+ * info, there is no way to confirm whether there is any standby
+ * configured for the known physical slots.
+ */
+ if (strcmp(*newval, "*") == 0)
+ {
+ GUC_check_errdetail("\"%s\" is not accepted for standby_slot_names",
+ *newval);
+ return false;
+ }
+
+ /* Now verify if the specified slots really exist and have correct type */
+ if (!validate_standby_slots(newval))
+ return false;
+
+ *extra = guc_strdup(ERROR, *newval);
+
+ return true;
+}
+
+/*
+ * GUC assign_hook for standby_slot_names
+ */
+void
+assign_standby_slot_names(const char *newval, void *extra)
+{
+ List *standby_slots;
+ MemoryContext oldcxt;
+ char *standby_slot_names_cpy = extra;
+
+ list_free(standby_slot_names_list);
+ standby_slot_names_list = NIL;
+
+ /* No value is specified for standby_slot_names. */
+ if (standby_slot_names_cpy == NULL)
+ return;
+
+ if (!SplitIdentifierString(standby_slot_names_cpy, ',', &standby_slots))
+ {
+ /* This should not happen if GUC checked check_standby_slot_names. */
+ elog(ERROR, "invalid list syntax");
+ }
+
+ /*
+ * Switch to the memory context under which GUC variables are allocated
+ * (GUCMemoryContext).
+ */
+ oldcxt = MemoryContextSwitchTo(GetMemoryChunkContext(standby_slot_names_cpy));
+ standby_slot_names_list = list_copy(standby_slots);
+ MemoryContextSwitchTo(oldcxt);
+}
+
+/*
+ * Return a copy of standby_slot_names_list if the copy flag is set to true,
+ * otherwise return the original list.
+ *
+ * Note that since we do not support syncing slots to cascading standbys, we
+ * return NIL if we are running in a standby to indicate that no standby slots
+ * need to be waited for, regardless of the copy flag value.
+ */
+List *
+GetStandbySlotList(bool copy)
+{
+ if (RecoveryInProgress())
+ return NIL;
+
+ if (copy)
+ return list_copy(standby_slot_names_list);
+ else
+ return standby_slot_names_list;
+}
+
+/*
+ * Filter the standby slots based on the specified log sequence number
+ * (wait_for_lsn).
+ *
+ * This function updates the passed standby_slots list, removing any slots that
+ * have already caught up to or surpassed the given wait_for_lsn.
+ */
+void
+FilterStandbySlots(XLogRecPtr wait_for_lsn, List **standby_slots)
+{
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, *standby_slots)
+ {
+ XLogRecPtr restart_lsn;
+ bool invalidated;
+ bool inactive;
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ /*
+ * It may happen that the slot specified in standby_slot_names GUC
+ * value is dropped, so let's skip over it.
+ */
+ ereport(WARNING,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not exist",
+ name, "standby_slot_names"));
+ continue;
+ }
+
+ if (SlotIsLogical(slot))
+ {
+ /*
+ * If a logical slot name is provided in standby_slot_names, issue
+ * a WARNING and skip it. Although logical slots are disallowed in
+ * the GUC check_hook(validate_standby_slots), it is still
+ * possible for a user to drop an existing physical slot and
+ * recreate a logical slot with the same name.
+ */
+ ereport(WARNING,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot have logical replication slot \"%s\" in parameter %s",
+ name, "standby_slot_names"));
+ continue;
+ }
+
+ SpinLockAcquire(&slot->mutex);
+ restart_lsn = slot->data.restart_lsn;
+ invalidated = slot->data.invalidated != RS_INVAL_NONE;
+ inactive = slot->active_pid == 0;
+ SpinLockRelease(&slot->mutex);
+
+ if (invalidated)
+ {
+ /* Specified physical slot has been invalidated */
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("physical slot \"%s\" specified in parameter %s has been invalidated",
+ name, "standby_slot_names"));
+ continue;
+ }
+
+ if (XLogRecPtrIsInvalid(restart_lsn) || restart_lsn < wait_for_lsn)
+ {
+ /* Log warning if no active_pid for this physical slot */
+ if (inactive)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not have active_pid",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider starting standby associated with \"%s\" or amend standby_slot_names.",
+ name));
+
+ /* Continue if the current slot hasn't caught up. */
+ continue;
+ }
+
+ Assert(restart_lsn >= wait_for_lsn);
+ *standby_slots = foreach_delete_current(*standby_slots, name);
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+}
+
+/*
+ * Wait for physical standby to confirm receiving the given lsn.
+ *
+ * Used by logical decoding SQL functions that acquired failover enabled slot.
+ * It waits for physical standbys corresponding to the physical slots specified
+ * in the standby_slot_names GUC.
+ */
+void
+WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
+{
+ List *standby_slots;
+
+ if (!MyReplicationSlot->data.failover)
+ return;
+
+ standby_slots = GetStandbySlotList(true);
+
+ if (standby_slots == NIL)
+ return;
+
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+
+ for (;;)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ list_free(standby_slots);
+ standby_slots = GetStandbySlotList(true);
+ }
+
+ FilterStandbySlots(wait_for_lsn, &standby_slots);
+
+ /* Exit if done waiting for every slot. */
+ if (standby_slots == NIL)
+ break;
+
+ /*
+ * We wait for the slots in the standby_slot_names to catch up, but we
+ * use a timeout (1s) so we can also check the if the
+ * standby_slot_names has been changed.
+ */
+ ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000,
+ WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION);
+ }
+
+ ConditionVariableCancelSleep();
+ list_free(standby_slots);
+}
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 768a304723..09067febe1 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -464,6 +464,8 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* crash, but this makes the data consistent after a clean shutdown.
*/
ReplicationSlotMarkDirty();
+
+ PhysicalWakeupLogicalWalSnd();
}
return retlsn;
@@ -504,6 +506,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto)
.segment_close = wal_segment_close),
NULL, NULL, NULL);
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to moveto lsn.
+ */
+ WaitForStandbyConfirmation(moveto);
+
/*
* Start reading at the slot's restart_lsn, which we know to point to
* a valid record.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 13bc3e0aee..1272237ae8 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1728,27 +1728,77 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
ProcessPendingWrites();
}
+/*
+ * Wake up the logical walsender processes with failover enabled slots if the
+ * currently acquired physical slot is specified in standby_slot_names GUC.
+ */
+void
+PhysicalWakeupLogicalWalSnd(void)
+{
+ ListCell *lc;
+ List *standby_slots;
+
+ Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot));
+
+ standby_slots = GetStandbySlotList(false);
+
+ foreach(lc, standby_slots)
+ {
+ char *name = lfirst(lc);
+
+ if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0)
+ {
+ ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv);
+ return;
+ }
+ }
+}
+
/*
* Wait till WAL < loc is flushed to disk so it can be safely sent to client.
*
- * Returns end LSN of flushed WAL. Normally this will be >= loc, but
- * if we detect a shutdown request (either from postmaster or client)
- * we will return early, so caller must always check.
+ * If the walsender holds a logical slot that has enabled failover, we also
+ * wait for all the specified streaming replication standby servers to
+ * confirm receipt of WAL up to RecentFlushPtr.
+ *
+ * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we
+ * detect a shutdown request (either from postmaster or client) we will return
+ * early, so caller must always check.
*/
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
int wakeEvents;
+ bool wait_for_standby = false;
+ uint32 wait_event;
+ List *standby_slots = NIL;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
+ if (MyReplicationSlot->data.failover && replication_active)
+ standby_slots = GetStandbySlotList(true);
+
/*
- * Fast path to avoid acquiring the spinlock in case we already know we
- * have enough WAL available. This is particularly interesting if we're
- * far behind.
+ * Check if all the standby servers have confirmed receipt of WAL up to
+ * RecentFlushPtr even when we already know we have enough WAL available.
+ *
+ * Note that we cannot directly return without checking the status of
+ * standby servers because the standby_slot_names may have changed, which
+ * means there could be new standby slots in the list that have not yet
+ * caught up to the RecentFlushPtr.
*/
- if (RecentFlushPtr != InvalidXLogRecPtr &&
- loc <= RecentFlushPtr)
- return RecentFlushPtr;
+ if (!XLogRecPtrIsInvalid(RecentFlushPtr) && loc <= RecentFlushPtr)
+ {
+ FilterStandbySlots(RecentFlushPtr, &standby_slots);
+
+ /*
+ * Fast path to avoid acquiring the spinlock in case we already know
+ * we have enough WAL available and all the standby servers have
+ * confirmed receipt of WAL up to RecentFlushPtr. This is particularly
+ * interesting if we're far behind.
+ */
+ if (standby_slots == NIL)
+ return RecentFlushPtr;
+ }
/* Get a more recent flush pointer. */
if (!RecoveryInProgress())
@@ -1770,6 +1820,13 @@ WalSndWaitForWal(XLogRecPtr loc)
{
ConfigReloadPending = false;
ProcessConfigFile(PGC_SIGHUP);
+
+ if (MyReplicationSlot->data.failover && replication_active)
+ {
+ list_free(standby_slots);
+ standby_slots = GetStandbySlotList(true);
+ }
+
SyncRepInitConfig();
}
@@ -1784,8 +1841,18 @@ WalSndWaitForWal(XLogRecPtr loc)
if (got_STOPPING)
XLogBackgroundFlush();
+ /*
+ * Update the standby slots that have not yet caught up to the flushed
+ * position. It is good to wait up to RecentFlushPtr and then let it
+ * send the changes to logical subscribers one by one which are
+ * already covered in RecentFlushPtr without needing to wait on every
+ * change for standby confirmation.
+ */
+ if (wait_for_standby)
+ FilterStandbySlots(RecentFlushPtr, &standby_slots);
+
/* Update our idea of the currently flushed position. */
- if (!RecoveryInProgress())
+ else if (!RecoveryInProgress())
RecentFlushPtr = GetFlushRecPtr(NULL);
else
RecentFlushPtr = GetXLogReplayRecPtr(NULL);
@@ -1813,9 +1880,18 @@ WalSndWaitForWal(XLogRecPtr loc)
!waiting_for_ping_response)
WalSndKeepalive(false, InvalidXLogRecPtr);
- /* check whether we're done */
- if (loc <= RecentFlushPtr)
+ if (loc > RecentFlushPtr)
+ wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
+ else if (standby_slots)
+ {
+ wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
+ wait_for_standby = true;
+ }
+ else
+ {
+ /* Already caught up and doesn't need to wait for standby_slots. */
break;
+ }
/* Waiting for new WAL. Since we need to wait, we're now caught up. */
WalSndCaughtUp = true;
@@ -1855,9 +1931,11 @@ WalSndWaitForWal(XLogRecPtr loc)
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL);
+ WalSndWait(wakeEvents, sleeptime, wait_event);
}
+ list_free(standby_slots);
+
/* reactivate latch so WalSndLoop knows to continue */
SetLatch(MyLatch);
return RecentFlushPtr;
@@ -2265,6 +2343,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn)
{
ReplicationSlotMarkDirty();
ReplicationSlotsComputeRequiredLSN();
+ PhysicalWakeupLogicalWalSnd();
}
/*
@@ -3538,6 +3617,7 @@ WalSndShmemInit(void)
ConditionVariableInit(&WalSndCtl->wal_flush_cv);
ConditionVariableInit(&WalSndCtl->wal_replay_cv);
+ ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv);
}
}
@@ -3607,8 +3687,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event)
*
* And, we use separate shared memory CVs for physical and logical
* walsenders for selective wake ups, see WalSndWakeup() for more details.
+ *
+ * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV
+ * until awakened by physical walsenders after the walreceiver confirms
+ * the receipt of the LSN.
*/
- if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
+ if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+ else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv);
else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 4fffb46625..2d1c700543 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP
LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server."
LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server."
SSL_OPEN_SERVER "Waiting for SSL while attempting connection."
+WAIT_FOR_STANDBY_CONFIRMATION "Waiting for the WAL to be received by physical standby."
WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process."
WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 527a2b2734..2eea26b5d5 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4639,6 +4639,20 @@ struct config_string ConfigureNamesString[] =
check_debug_io_direct, assign_debug_io_direct, NULL
},
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("Lists streaming replication standby server slot "
+ "names that logical WAL sender processes will wait for."),
+ gettext_noop("Decoded changes are sent out to plugins by logical "
+ "WAL sender processes only after specified "
+ "replication slots confirm receiving WAL."),
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, assign_standby_slot_names, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c97f9a25f0..cadfe10958 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -334,6 +334,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsender processes will wait for
# - Standby Servers -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index acbf567150..04a2717138 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -226,6 +226,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
+extern PGDLLIMPORT char *standby_slot_names;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -274,4 +275,9 @@ extern void CheckSlotPermissions(void);
extern ReplicationSlotInvalidationCause
GetSlotInvalidationCause(const char *conflict_reason);
+extern List *GetStandbySlotList(bool copy);
+extern void FilterStandbySlots(XLogRecPtr wait_for_lsn,
+ List **standby_slots);
+extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn);
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index 0c3996e926..f2d8297f01 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -39,6 +39,7 @@ extern void InitWalSender(void);
extern bool exec_replication_command(const char *cmd_string);
extern void WalSndErrorCleanup(void);
extern void WalSndResourceCleanup(bool isCommit);
+extern void PhysicalWakeupLogicalWalSnd(void);
extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
extern void WalSndSignals(void);
extern Size WalSndShmemSize(void);
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 3113e9ea47..3c134c8edf 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -113,6 +113,13 @@ typedef struct
ConditionVariable wal_flush_cv;
ConditionVariable wal_replay_cv;
+ /*
+ * Used by physical walsenders holding slots specified in
+ * standby_slot_names to wake up logical walsenders holding failover
+ * enabled slots when a walreceiver confirms the receipt of LSN.
+ */
+ ConditionVariable wal_confirm_rcv_cv;
+
WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER];
} WalSndCtlData;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 339c490300..dcf9a040d1 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -163,5 +163,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra,
extern void assign_wal_consistency_checking(const char *newval, void *extra);
extern bool check_wal_segment_size(int *newval, void **extra, GucSource source);
extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
+extern bool check_standby_slot_names(char **newval, void **extra,
+ GucSource source);
+extern void assign_standby_slot_names(const char *newval, void *extra);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl
index 5c7b4ca5e3..85f019774c 100644
--- a/src/test/recovery/t/006_logical_decoding.pl
+++ b/src/test/recovery/t/006_logical_decoding.pl
@@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'},
undef, 'logical slot was actually dropped with DB');
# Test logical slot advancing and its durability.
+# Pass failover=true (last-arg), it should not have any impact on advancing.
my $logical_slot = 'logical_slot';
$node_primary->safe_psql('postgres',
- "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);"
+ "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);"
);
$node_primary->psql(
'postgres', "
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 968aa7b05b..53480f5acf 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -446,11 +446,195 @@ ok( $standby1->poll_query_until(
"SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot' AND synced AND NOT temporary;"),
'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby');
+##################################################
+# Test primary disallowing specified logical replication slots getting ahead of
+# specified physical replication slots. It uses the following set up:
+#
+# | ----> standby1 (primary_slot_name = sb1_slot)
+# | ----> standby2 (primary_slot_name = sb2_slot)
+# primary ----- |
+# | ----> subscriber1 (failover = true)
+# | ----> subscriber2 (failover = false)
+#
+# standby_slot_names = 'sb1_slot'
+#
+# Set up is configured in such a way that the logical slot of subscriber1 is
+# enabled failover, thus it will wait for the physical slot of
+# standby1(sb1_slot) to catch up before sending decoded changes to subscriber1.
+##################################################
+
+$backup_name = 'backup3';
+
+$primary->psql('postgres',
+ q{SELECT pg_create_physical_replication_slot('sb2_slot');});
+
+$primary->backup($backup_name);
+
+# Create another standby
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup(
+ $primary, $backup_name,
+ has_streaming => 1,
+ has_restoring => 1);
+$standby2->append_conf(
+ 'postgresql.conf', qq(
+primary_slot_name = 'sb2_slot'
+));
+$standby2->start;
+$primary->wait_for_replay_catchup($standby2);
+
+# Configure primary to disallow any logical slots that enabled failover from
+# getting ahead of specified physical replication slot (sb1_slot).
+$primary->append_conf(
+ 'postgresql.conf', qq(
+standby_slot_names = 'sb1_slot'
+));
+$primary->reload;
+
+# Create another subscriber node without enabling failover, wait for sync to
+# complete
+my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$subscriber2->init;
+$subscriber2->start;
+$subscriber2->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE tab_int (a int PRIMARY KEY);
+ CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot);
+]);
+
+$subscriber1->wait_for_subscription_sync;
+
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+# Stop the standby associated with the specified physical replication slot so
+# that the logical replication slot won't receive changes until the standby
+# comes up.
+$standby1->stop;
+
+# Create some data on the primary
+my $primary_row_count = 20;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(11, $primary_row_count);");
+
+# Wait for the standby that's up and running gets the data from primary
+$primary->wait_for_replay_catchup($standby2);
+$result = $standby2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby2 gets data from primary");
+
+# Wait for the subscription that's up and running and is not enabled for failover.
+# It gets the data from primary without waiting for any standbys.
+$primary->wait_for_catchup('regress_mysub2');
+$result = $subscriber2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "subscriber2 gets data from primary");
+
+# The subscription that's up and running and is enabled for failover
+# doesn't get the data from primary and keeps waiting for the
+# standby specified in standby_slot_names.
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) <> $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data from primary until standby1 acknowledges changes"
+);
+
+# Start the standby specified in standby_slot_names and wait for it to catch
+# up with the primary.
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+$result = $standby1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby1 gets data from primary");
+
+# Now that the standby specified in standby_slot_names is up and running,
+# primary must send the decoded changes to subscription enabled for failover
+# While the standby was down, this subscriber didn't receive any data from
+# primary i.e. the primary didn't allow it to go ahead of standby.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 acknowledges changes");
+
+# Stop the standby associated with the specified physical replication slot so
+# that the logical replication slot won't receive changes until the standby
+# slot's restart_lsn is advanced or the slot is removed from the
+# standby_slot_names list.
+$primary->safe_psql('postgres', "TRUNCATE tab_int;");
+$primary->wait_for_catchup('regress_mysub1');
+$standby1->stop;
+
+##################################################
+# Verify that when using pg_logical_slot_get_changes to consume changes from a
+# logical slot with failover enabled, it will also wait for the slots specified
+# in standby_slot_names to catch up.
+##################################################
+
+# Create a logical 'test_decoding' replication slot with failover enabled
+$primary->safe_psql('postgres',
+ "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);"
+);
+
+my $back_q = $primary->background_psql('postgres', on_error_stop => 0);
+my $pid = $back_q->query('SELECT pg_backend_pid()');
+
+# Try and get changes from the logical slot with failover enabled.
+my $offset = -s $primary->logfile;
+$back_q->query_until(qr//,
+ "SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);\n");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+ok($primary->safe_psql('postgres', "SELECT pg_cancel_backend($pid)"),
+ "cancelling pg_logical_slot_get_changes command");
+
+$back_q->quit;
+
+$primary->safe_psql('postgres',
+ "SELECT pg_drop_replication_slot('test_slot');"
+);
+
+##################################################
+# Test that logical replication will wait for the user-created inactive
+# physical slot to catch up until we remove the slot from standby_slot_names.
+##################################################
+
+# Create some data on the primary
+$primary_row_count = 10;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);");
+
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data as the sb1_slot doesn't catch up");
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the primary server should now
+# send the decoded changes to the subscription.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list"
+);
+
##################################################
# Promote the standby1 to primary. Confirm that:
# a) the slot 'lsub1_slot' is retained on the new primary
# b) logical replication for regress_mysub1 is resumed successfully after failover
##################################################
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+
$standby1->promote;
# Update subscription with the new primary's connection info
--
2.30.0.windows.2
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-02-27 07:17 Peter Smith <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 3 replies; 68+ messages in thread
From: Peter Smith @ 2024-02-27 07:17 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
Here are some review comments for v99-0001
==========
0. GENERAL.
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsender processes will wait for
IMO the GUC name is too generic. There is nothing in this name to
suggest it has anything to do with logical slot synchronization; that
meaning is only found in the accompanying comment -- it would be
better if the GUC name itself were more self-explanatory.
e.g. Maybe like 'wal_sender_sync_standby_slot_names' or
'wal_sender_standby_slot_names', 'wal_sender_wait_for_standby_slots',
or ...
(Of course, this has some impact across docs and comments and
variables in the patch).
==========
Commit Message
1.
A new parameter named standby_slot_names is introduced.
Maybe quote the GUC names here to make it more readable.
~~
2.
Additionally, The SQL functions pg_logical_slot_get_changes and
pg_replication_slot_advance are modified to wait for the replication slots
mentioned in standby_slot_names to catch up before returning the changes
to the user.
~
2a.
"pg_replication_slot_advance" is a typo? Did you mean
pg_logical_replication_slot_advance?
~
2b.
The "before returning the changes to the user" seems like it is
referring only to the first function.
Maybe needs slight rewording like:
/before returning the changes to the user./ before returning./
==========
doc/src/sgml/config.sgml
3. standby_slot_names
+ <para>
+ List of physical slots guarantees that logical replication slots with
+ failover enabled do not consume changes until those changes
are received
+ and flushed to corresponding physical standbys. If a logical
replication
+ connection is meant to switch to a physical standby after the
standby is
+ promoted, the physical replication slot for the standby
should be listed
+ here. Note that logical replication will not proceed if the slots
+ specified in the standby_slot_names do not exist or are invalidated.
+ </para>
The wording doesn't seem right. IMO this should be worded much like
how this GUC is described in guc_tables.c
e.g something a bit like:
Lists the streaming replication standby server slot names that logical
WAL sender processes will wait for. Logical WAL sender processes will
send decoded changes to plugins only after the specified replication
slots confirm receiving WAL. This guarantees that logical replication
slots with failover enabled do not consume changes until those changes
are received and flushed to corresponding physical standbys...
==========
doc/src/sgml/logicaldecoding.sgml
4. Section 48.2.3 Replication Slot Synchronization
+ It's also highly recommended that the said physical replication slot
+ is named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ list on the primary, to prevent the subscriber from consuming changes
+ faster than the hot standby. But once we configure it, then
certain latency
+ is expected in sending changes to logical subscribers due to wait on
+ physical replication slots in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
4a.
/It's also highly/It is highly/
~
4b.
BEFORE
But once we configure it, then certain latency is expected in sending
changes to logical subscribers due to wait on physical replication
slots in <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
SUGGESTION
Even when correctly configured, some latency is expected when sending
changes to logical subscribers due to the waiting on slots named in
standby_slot_names.
==========
.../replication/logical/logicalfuncs.c
5. pg_logical_slot_get_changes_guts
+ if (XLogRecPtrIsInvalid(upto_lsn))
+ wait_for_wal_lsn = end_of_wal;
+ else
+ wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
+
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to wait_for_wal_lsn.
+ */
+ WaitForStandbyConfirmation(wait_for_wal_lsn);
Perhaps those statements all belong together with the comment up-front. e.g.
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to wait_for_wal_lsn.
+ */
+ if (XLogRecPtrIsInvalid(upto_lsn))
+ wait_for_wal_lsn = end_of_wal;
+ else
+ wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
+ WaitForStandbyConfirmation(wait_for_wal_lsn);
==========
src/backend/replication/logical/slotsync.c
==========
src/backend/replication/slot.c
6.
+static bool
+validate_standby_slots(char **newval)
+{
+ char *rawname;
+ List *elemlist;
+ ListCell *lc;
+ bool ok;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Verify syntax and parse string into a list of identifiers */
+ ok = SplitIdentifierString(rawname, ',', &elemlist);
+
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
+
+ /*
+ * If the replication slots' data have been initialized, verify if the
+ * specified slots exist and are logical slots.
+ */
+ else if (ReplicationSlotCtl)
+ {
+ foreach(lc, elemlist)
6a.
So, if the ReplicationSlotCtl is NULL, it is possible to return
ok=true without ever checking if the slots exist or are of the correct
kind. I am wondering what are the ramifications of that. -- e.g.
assuming names are OK when maybe they aren't OK at all. AFAICT this
works because it relies on getting subsequent WARNINGS when calling
FilterStandbySlots(). If that is correct then maybe the comment here
can be enhanced to say so.
Indeed, if it works like that, now I am wondering do we need this for
loop validation at all. e.g. it seems just a matter of timing whether
we get ERRORs validating the GUC here, or WARNINGS later in the
FilterStandbySlots. Maybe we don't need the double-checking and it is
enough to check in FilterStandbySlots?
~
6b.
AFAIK there are alternative foreach macros available now, so you
shouldn't need to declare the ListCell.
~~~
7. check_standby_slot_names
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ if (strcmp(*newval, "") == 0)
+ return true;
Using strcmp seems like an overkill way to check for empty string.
SUGGESTION
if (*newval == '\0')
return true;
~~~
8.
+ if (strcmp(*newval, "*") == 0)
+ {
+ GUC_check_errdetail("\"%s\" is not accepted for standby_slot_names",
+ *newval);
+ return false;
+ }
It seems overkill to use a format specifier when "*" is already the known value.
SUGGESTION
GUC_check_errdetail("Wildcard \"*\" is not accepted for standby_slot_names.");
~~~
9.
+ /* Now verify if the specified slots really exist and have correct type */
+ if (!validate_standby_slots(newval))
+ return false;
As in a prior comment, if ReplicationSlotCtl is NULL then it is not
always going to do exactly what that comment says it is doing...
~~~
10. assign_standby_slot_names
+ if (!SplitIdentifierString(standby_slot_names_cpy, ',', &standby_slots))
+ {
+ /* This should not happen if GUC checked check_standby_slot_names. */
+ elog(ERROR, "invalid list syntax");
+ }
I didn't see how it is possible to get here without having first
executed check_standby_slot_names. But, if it can happen, then maybe
describe the scenario in the comment.
~~~
11.
+ * Note that since we do not support syncing slots to cascading standbys, we
+ * return NIL if we are running in a standby to indicate that no standby slots
+ * need to be waited for, regardless of the copy flag value.
I didn't understand the relevance of even mentioning "regardless of
the copy flag value".
~~~
12. FilterStandbySlots
+ errhint("Consider starting standby associated with \"%s\" or amend
standby_slot_names.",
+ name));
This errhint should use a format substitution for the GUC
"standby_slot_names" for consistency with everything else.
~~~
13. WaitForStandbyConfirmation
+ /*
+ * We wait for the slots in the standby_slot_names to catch up, but we
+ * use a timeout (1s) so we can also check the if the
+ * standby_slot_names has been changed.
+ */
+ ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000,
+ WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION);
Typo "the if the"
==========
src/backend/replication/slotfuncs.c
14. pg_physical_replication_slot_advance
+
+ PhysicalWakeupLogicalWalSnd();
Should this have a comment to say what it is for?
==========
src/backend/replication/walsender.c
15.
+/*
+ * Wake up the logical walsender processes with failover enabled slots if the
+ * currently acquired physical slot is specified in standby_slot_names GUC.
+ */
+void
+PhysicalWakeupLogicalWalSnd(void)
+{
+ ListCell *lc;
+ List *standby_slots;
+
+ Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot));
+
+ standby_slots = GetStandbySlotList(false);
+
+ foreach(lc, standby_slots)
+ {
+ char *name = lfirst(lc);
+
+ if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0)
+ {
+ ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv);
+ return;
+ }
+ }
+}
15a.
There already exists another function called WalSndWakeup(bool
physical, bool logical), so I think this new one should use a similar
name pattern -- e.g. maybe like WalSndWakeupLogicalForSlotSync or ...
~
15b.
IIRC there are some new List macros you can use instead of needing to
declare the ListCell?
==========
.../utils/activity/wait_event_names.txt
16.
+WAIT_FOR_STANDBY_CONFIRMATION "Waiting for the WAL to be received by
physical standby."
Moving the 'the' will make this more consistent with all other
"Waiting for WAL..." names.
SUGGESTION
Waiting for WAL to be received by the physical standby.
==========
src/backend/utils/misc/guc_tables.c
17.
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("Lists streaming replication standby server slot "
+ "names that logical WAL sender processes will wait for."),
+ gettext_noop("Decoded changes are sent out to plugins by logical "
+ "WAL sender processes only after specified "
+ "replication slots confirm receiving WAL."),
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, assign_standby_slot_names, NULL
+ },
The wording of the detail msg feels kind of backwards to me.
BEFORE
Decoded changes are sent out to plugins by logical WAL sender
processes only after specified replication slots confirm receiving
WAL.
SUGGESTION
Logical WAL sender processes will send decoded changes to plugins only
after the specified replication slots confirm receiving WAL.
==========
src/backend/utils/misc/postgresql.conf.sample
18.
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsender processes will wait for
I'm not sure this is the best GUC name. See the general comment #0
above in this post.
==========
src/include/replication/slot.h
==========
src/include/replication/walsender.h
==========
src/include/replication/walsender_private.h
==========
src/include/utils/guc_hooks.h
==========
src/test/recovery/t/006_logical_decoding.pl
19.
+# Pass failover=true (last-arg), it should not have any impact on advancing.
SUGGESTION
Passing failover=true (last arg) should not have any impact on advancing.
==========
.../t/040_standby_failover_slots_sync.pl
20.
+#
+# | ----> standby1 (primary_slot_name = sb1_slot)
+# | ----> standby2 (primary_slot_name = sb2_slot)
+# primary ----- |
+# | ----> subscriber1 (failover = true)
+# | ----> subscriber2 (failover = false)
In the diagram, the "--->" means a mixture of physical standbys and
logical pub/sub replication. Maybe it can be a bit clearer?
SUGGESTION
# primary (publisher)
#
# (physical standbys)
# | ----> standby1 (primary_slot_name = sb1_slot)
# | ----> standby2 (primary_slot_name = sb2_slot)
#
# (logical replication)
# | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
# | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
~~~
21.
+# Set up is configured in such a way that the logical slot of subscriber1 is
+# enabled failover, thus it will wait for the physical slot of
+# standby1(sb1_slot) to catch up before sending decoded changes to subscriber1.
/is enabled failover/is enabled for failover/
~~~
22.
+# Create another subscriber node without enabling failover, wait for sync to
+# complete
+my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$subscriber2->init;
+$subscriber2->start;
+$subscriber2->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE tab_int (a int PRIMARY KEY);
+ CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr'
PUBLICATION regress_mypub WITH (slot_name = lsub2_slot);
+]);
+
+$subscriber1->wait_for_subscription_sync;
+
Is this meant to wait for 'subscription2'?
~~~
23.
# Stop the standby associated with the specified physical replication slot so
# that the logical replication slot won't receive changes until the standby
# comes up.
Maybe this can give the values for better understanding:
SUGGESTION
Stop the standby associated with the specified physical replication
slot (sb1_slot) so that the logical replication slot (lsub1_slot)
won't receive changes until the standby comes up.
~~~
24.
+# Wait for the standby that's up and running gets the data from primary
SUGGESTION
Wait until the standby2 that's still running gets the data from the primary.
~~~
25.
+# Wait for the subscription that's up and running and is not enabled
for failover.
+# It gets the data from primary without waiting for any standbys.
SUGGESTION
Wait for subscription2 to get the data from the primary. This
subscription was not enabled for failover so it gets the data without
waiting for any standbys.
~~~
26.
+# The subscription that's up and running and is enabled for failover
+# doesn't get the data from primary and keeps waiting for the
+# standby specified in standby_slot_names.
SUGGESTION
The subscription1 was enabled for failover so it doesn't get the data
from primary and keeps waiting for the standby specified in
standby_slot_names (sb1_slot aka standby1).
~~~
27.
+# Start the standby specified in standby_slot_names and wait for it to catch
+# up with the primary.
SUGGESTION
Start the standby specified in standby_slot_names (sb1_slot aka
standby1) and wait for it to catch up with the primary.
~~~
28.
+# Now that the standby specified in standby_slot_names is up and running,
+# primary must send the decoded changes to subscription enabled for failover
+# While the standby was down, this subscriber didn't receive any data from
+# primary i.e. the primary didn't allow it to go ahead of standby.
SUGGESTION
Now that the standby specified in standby_slot_names is up and
running, the primary can send the decoded changes to the subscription
enabled for failover (i.e. subscription1). While the standby was down,
subscription1 didn't receive any data from the primary. i.e. the
primary didn't allow it to go ahead of standby.
~~~
29.
+# Stop the standby associated with the specified physical replication slot so
+# that the logical replication slot won't receive changes until the standby
+# slot's restart_lsn is advanced or the slot is removed from the
+# standby_slot_names list.
+$primary->safe_psql('postgres', "TRUNCATE tab_int;");
+$primary->wait_for_catchup('regress_mysub1');
+$standby1->stop;
Isn't this fragment more like the first step of the *next* TEST
instead of the last step of this one?
~~~
30.
+##################################################
+# Verify that when using pg_logical_slot_get_changes to consume changes from a
+# logical slot with failover enabled, it will also wait for the slots specified
+# in standby_slot_names to catch up.
+##################################################
AFAICT this test is checking only that the function cannot return
while waiting for the stopped standby, but it doesn't seem to check
that it *does* return when the stopped standby comes alive again.
~~~
31.
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data as the sb1_slot doesn't catch up");
Do you think this fragment should have a comment?
======
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-02-27 10:37 Bertrand Drouvot <[email protected]>
parent: Peter Smith <[email protected]>
2 siblings, 1 reply; 68+ messages in thread
From: Bertrand Drouvot @ 2024-02-27 10:37 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
Hi,
On Tue, Feb 27, 2024 at 06:17:44PM +1100, Peter Smith wrote:
> +static bool
> +validate_standby_slots(char **newval)
> +{
> + char *rawname;
> + List *elemlist;
> + ListCell *lc;
> + bool ok;
> +
> + /* Need a modifiable copy of string */
> + rawname = pstrdup(*newval);
> +
> + /* Verify syntax and parse string into a list of identifiers */
> + ok = SplitIdentifierString(rawname, ',', &elemlist);
> +
> + if (!ok)
> + {
> + GUC_check_errdetail("List syntax is invalid.");
> + }
> +
> + /*
> + * If the replication slots' data have been initialized, verify if the
> + * specified slots exist and are logical slots.
> + */
> + else if (ReplicationSlotCtl)
> + {
> + foreach(lc, elemlist)
>
> 6a.
> So, if the ReplicationSlotCtl is NULL, it is possible to return
> ok=true without ever checking if the slots exist or are of the correct
> kind. I am wondering what are the ramifications of that. -- e.g.
> assuming names are OK when maybe they aren't OK at all. AFAICT this
> works because it relies on getting subsequent WARNINGS when calling
> FilterStandbySlots(). If that is correct then maybe the comment here
> can be enhanced to say so.
>
> Indeed, if it works like that, now I am wondering do we need this for
> loop validation at all. e.g. it seems just a matter of timing whether
> we get ERRORs validating the GUC here, or WARNINGS later in the
> FilterStandbySlots. Maybe we don't need the double-checking and it is
> enough to check in FilterStandbySlots?
Good point, I have the feeling that it is enough to check in FilterStandbySlots().
Indeed, if the value is syntactically correct, then I think that its actual value
"really" matters when the logical decoding is starting/running, does it provide
additional benefits "before" that?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-02-27 11:59 Amit Kapila <[email protected]>
parent: Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 68+ messages in thread
From: Amit Kapila @ 2024-02-27 11:59 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Peter Smith <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Tue, Feb 27, 2024 at 4:07 PM Bertrand Drouvot
<[email protected]> wrote:
>
> On Tue, Feb 27, 2024 at 06:17:44PM +1100, Peter Smith wrote:
> > +static bool
> > +validate_standby_slots(char **newval)
> > +{
> > + char *rawname;
> > + List *elemlist;
> > + ListCell *lc;
> > + bool ok;
> > +
> > + /* Need a modifiable copy of string */
> > + rawname = pstrdup(*newval);
> > +
> > + /* Verify syntax and parse string into a list of identifiers */
> > + ok = SplitIdentifierString(rawname, ',', &elemlist);
> > +
> > + if (!ok)
> > + {
> > + GUC_check_errdetail("List syntax is invalid.");
> > + }
> > +
> > + /*
> > + * If the replication slots' data have been initialized, verify if the
> > + * specified slots exist and are logical slots.
> > + */
> > + else if (ReplicationSlotCtl)
> > + {
> > + foreach(lc, elemlist)
> >
> > 6a.
> > So, if the ReplicationSlotCtl is NULL, it is possible to return
> > ok=true without ever checking if the slots exist or are of the correct
> > kind. I am wondering what are the ramifications of that. -- e.g.
> > assuming names are OK when maybe they aren't OK at all. AFAICT this
> > works because it relies on getting subsequent WARNINGS when calling
> > FilterStandbySlots(). If that is correct then maybe the comment here
> > can be enhanced to say so.
> >
> > Indeed, if it works like that, now I am wondering do we need this for
> > loop validation at all. e.g. it seems just a matter of timing whether
> > we get ERRORs validating the GUC here, or WARNINGS later in the
> > FilterStandbySlots. Maybe we don't need the double-checking and it is
> > enough to check in FilterStandbySlots?
>
> Good point, I have the feeling that it is enough to check in FilterStandbySlots().
>
I think it is better if we get earlier in a case where the parameter
is changed and performed SIGHUP instead of waiting till we get to
logical decoding. So, there is merit in keeping these checks during
initial validation.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-02-27 12:35 Amit Kapila <[email protected]>
parent: Peter Smith <[email protected]>
2 siblings, 1 reply; 68+ messages in thread
From: Amit Kapila @ 2024-02-27 12:35 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Tue, Feb 27, 2024 at 12:48 PM Peter Smith <[email protected]> wrote:
>
> Here are some review comments for v99-0001
>
> ==========
> 0. GENERAL.
>
> +#standby_slot_names = '' # streaming replication standby server slot names that
> + # logical walsender processes will wait for
>
> IMO the GUC name is too generic. There is nothing in this name to
> suggest it has anything to do with logical slot synchronization; that
> meaning is only found in the accompanying comment -- it would be
> better if the GUC name itself were more self-explanatory.
>
> e.g. Maybe like 'wal_sender_sync_standby_slot_names' or
> 'wal_sender_standby_slot_names', 'wal_sender_wait_for_standby_slots',
> or ...
>
It would be wrong and or misleading to append wal_sender to this GUC
name as this is used during SQL APIs as well. Also, adding wait sounds
more like a boolean. So, I don't see the proposed names any better
than the current one.
> ~~~
>
> 9.
> + /* Now verify if the specified slots really exist and have correct type */
> + if (!validate_standby_slots(newval))
> + return false;
>
> As in a prior comment, if ReplicationSlotCtl is NULL then it is not
> always going to do exactly what that comment says it is doing...
>
It will do what the comment says when invoked as part of the SIGHUP
signal. I think the current comment is okay.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-02-28 02:23 Zhijie Hou (Fujitsu) <[email protected]>
parent: Peter Smith <[email protected]>
2 siblings, 2 replies; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-02-28 02:23 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Tuesday, February 27, 2024 3:18 PM Peter Smith <[email protected]> wrote:
>
> Here are some review comments for v99-0001
Thanks for the comments!
> Commit Message
>
> 1.
> A new parameter named standby_slot_names is introduced.
>
> Maybe quote the GUC names here to make it more readable.
Added.
>
> ~~
>
> 2.
> Additionally, The SQL functions pg_logical_slot_get_changes and
> pg_replication_slot_advance are modified to wait for the replication slots
> mentioned in standby_slot_names to catch up before returning the changes to
> the user.
>
> ~
>
> 2a.
> "pg_replication_slot_advance" is a typo? Did you mean
> pg_logical_replication_slot_advance?
pg_logical_replication_slot_advance is not a user visible function. So the
pg_replication_slot_advance is correct.
>
> ~
>
> 2b.
> The "before returning the changes to the user" seems like it is referring only to
> the first function.
>
> Maybe needs slight rewording like:
> /before returning the changes to the user./ before returning./
Changed.
>
> ==========
> doc/src/sgml/config.sgml
>
> 3. standby_slot_names
>
> + <para>
> + List of physical slots guarantees that logical replication slots with
> + failover enabled do not consume changes until those changes
> are received
> + and flushed to corresponding physical standbys. If a logical
> replication
> + connection is meant to switch to a physical standby after the
> standby is
> + promoted, the physical replication slot for the standby
> should be listed
> + here. Note that logical replication will not proceed if the slots
> + specified in the standby_slot_names do not exist or are invalidated.
> + </para>
>
> The wording doesn't seem right. IMO this should be worded much like how this
> GUC is described in guc_tables.c
>
> e.g something a bit like:
>
> Lists the streaming replication standby server slot names that logical WAL
> sender processes will wait for. Logical WAL sender processes will send
> decoded changes to plugins only after the specified replication slots confirm
> receiving WAL. This guarantees that logical replication slots with failover
> enabled do not consume changes until those changes are received and flushed
> to corresponding physical standbys...
Changed.
>
> ==========
> doc/src/sgml/logicaldecoding.sgml
>
> 4. Section 48.2.3 Replication Slot Synchronization
>
> + It's also highly recommended that the said physical replication slot
> + is named in
> + <link
> linkend="guc-standby-slot-names"><varname>standby_slot_names</varna
> me></link>
> + list on the primary, to prevent the subscriber from consuming changes
> + faster than the hot standby. But once we configure it, then
> certain latency
> + is expected in sending changes to logical subscribers due to wait on
> + physical replication slots in
> + <link
> +
> linkend="guc-standby-slot-names"><varname>standby_slot_names</varna
> me>
> + </link>
>
> 4a.
> /It's also highly/It is highly/
>
> ~
>
> 4b.
>
> BEFORE
> But once we configure it, then certain latency is expected in sending changes
> to logical subscribers due to wait on physical replication slots in <link
> linkend="guc-standby-slot-names"><varname>standby_slot_names</varna
> me></link>
>
> SUGGESTION
> Even when correctly configured, some latency is expected when sending
> changes to logical subscribers due to the waiting on slots named in
> standby_slot_names.
Changed.
>
> ==========
> .../replication/logical/logicalfuncs.c
>
> 5. pg_logical_slot_get_changes_guts
>
> + if (XLogRecPtrIsInvalid(upto_lsn))
> + wait_for_wal_lsn = end_of_wal;
> + else
> + wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
> +
> + /*
> + * Wait for specified streaming replication standby servers (if any)
> + * to confirm receipt of WAL up to wait_for_wal_lsn.
> + */
> + WaitForStandbyConfirmation(wait_for_wal_lsn);
>
> Perhaps those statements all belong together with the comment up-front. e.g.
>
> + /*
> + * Wait for specified streaming replication standby servers (if any)
> + * to confirm receipt of WAL up to wait_for_wal_lsn.
> + */
> + if (XLogRecPtrIsInvalid(upto_lsn))
> + wait_for_wal_lsn = end_of_wal;
> + else
> + wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
> + WaitForStandbyConfirmation(wait_for_wal_lsn);
Changed.
>
> ==========
> src/backend/replication/logical/slotsync.c
>
> ==========
> src/backend/replication/slot.c
>
> 6.
> +static bool
> +validate_standby_slots(char **newval)
> +{
> + char *rawname;
> + List *elemlist;
> + ListCell *lc;
> + bool ok;
> +
> + /* Need a modifiable copy of string */ rawname = pstrdup(*newval);
> +
> + /* Verify syntax and parse string into a list of identifiers */ ok =
> + SplitIdentifierString(rawname, ',', &elemlist);
> +
> + if (!ok)
> + {
> + GUC_check_errdetail("List syntax is invalid."); }
> +
> + /*
> + * If the replication slots' data have been initialized, verify if the
> + * specified slots exist and are logical slots.
> + */
> + else if (ReplicationSlotCtl)
> + {
> + foreach(lc, elemlist)
>
> 6a.
> So, if the ReplicationSlotCtl is NULL, it is possible to return ok=true without
> ever checking if the slots exist or are of the correct kind. I am wondering what
> are the ramifications of that. -- e.g.
> assuming names are OK when maybe they aren't OK at all. AFAICT this works
> because it relies on getting subsequent WARNINGS when calling
> FilterStandbySlots(). If that is correct then maybe the comment here can be
> enhanced to say so.
>
> Indeed, if it works like that, now I am wondering do we need this for loop
> validation at all. e.g. it seems just a matter of timing whether we get ERRORs
> validating the GUC here, or WARNINGS later in the FilterStandbySlots. Maybe
> we don't need the double-checking and it is enough to check in
> FilterStandbySlots?
I think the check is OK so didn’t change this.
>
> ~
>
> 6b.
> AFAIK there are alternative foreach macros available now, so you shouldn't
> need to declare the ListCell.
Changed.
>
> ~~~
>
> 7. check_standby_slot_names
>
> +bool
> +check_standby_slot_names(char **newval, void **extra, GucSource source)
> +{ if (strcmp(*newval, "") == 0) return true;
>
> Using strcmp seems like an overkill way to check for empty string.
>
> SUGGESTION
>
> if (*newval == '\0')
> return true;
>
Changed.
> ~~~
>
> 8.
> + if (strcmp(*newval, "*") == 0)
> + {
> + GUC_check_errdetail("\"%s\" is not accepted for standby_slot_names",
> + *newval); return false; }
>
> It seems overkill to use a format specifier when "*" is already the known value.
>
> SUGGESTION
> GUC_check_errdetail("Wildcard \"*\" is not accepted for
> standby_slot_names.");
>
Changed.
> ~~~
>
> 9.
> + /* Now verify if the specified slots really exist and have correct
> + type */ if (!validate_standby_slots(newval)) return false;
>
> As in a prior comment, if ReplicationSlotCtl is NULL then it is not always going
> to do exactly what that comment says it is doing...
I think the comment is OK, one can check the detail in the
function definition if needed.
>
> ~~~
>
> 10. assign_standby_slot_names
>
> + if (!SplitIdentifierString(standby_slot_names_cpy, ',',
> + &standby_slots)) {
> + /* This should not happen if GUC checked check_standby_slot_names. */
> + elog(ERROR, "invalid list syntax"); }
>
> I didn't see how it is possible to get here without having first executed
> check_standby_slot_names. But, if it can happen, then maybe describe the
> scenario in the comment.
This is sanity check which we don't expect to happen, which follows similar style of preprocessNamespacePath.
>
> ~~~
>
> 11.
> + * Note that since we do not support syncing slots to cascading
> + standbys, we
> + * return NIL if we are running in a standby to indicate that no
> + standby slots
> + * need to be waited for, regardless of the copy flag value.
>
> I didn't understand the relevance of even mentioning "regardless of the copy
> flag value".
Removed.
>
> ~~~
>
> 12. FilterStandbySlots
>
> + errhint("Consider starting standby associated with \"%s\" or amend
> standby_slot_names.",
> + name));
>
> This errhint should use a format substitution for the GUC "standby_slot_names"
> for consistency with everything else.
Changed.
>
> ~~~
>
> 13. WaitForStandbyConfirmation
>
> + /*
> + * We wait for the slots in the standby_slot_names to catch up, but we
> + * use a timeout (1s) so we can also check the if the
> + * standby_slot_names has been changed.
> + */
> + ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000,
> + WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION);
>
> Typo "the if the"
>
Changed.
> ==========
> src/backend/replication/slotfuncs.c
>
> 14. pg_physical_replication_slot_advance
> +
> + PhysicalWakeupLogicalWalSnd();
>
> Should this have a comment to say what it is for?
>
Added.
> ==========
> src/backend/replication/walsender.c
>
> 15.
> +/*
> + * Wake up the logical walsender processes with failover enabled slots
> +if the
> + * currently acquired physical slot is specified in standby_slot_names GUC.
> + */
> +void
> +PhysicalWakeupLogicalWalSnd(void)
> +{
> + ListCell *lc;
> + List *standby_slots;
> +
> + Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot));
> +
> + standby_slots = GetStandbySlotList(false);
> +
> + foreach(lc, standby_slots)
> + {
> + char *name = lfirst(lc);
> +
> + if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0) {
> +ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv);
> + return;
> + }
> + }
> +}
>
> 15a.
> There already exists another function called WalSndWakeup(bool physical,
> bool logical), so I think this new one should use a similar name pattern -- e.g.
> maybe like WalSndWakeupLogicalForSlotSync or ...
WalSndWakeup is a general function for both physical and logical sender, but
our new function is specific to physical sender which is more similar to
PhysicalConfirmReceivedLocation/ PhysicalReplicationSlotNewXmin, so I think the
current name is ok.
>
> ~
>
> 15b.
> IIRC there are some new List macros you can use instead of needing to declare
> the ListCell?
Changed.
>
> ==========
> .../utils/activity/wait_event_names.txt
>
> 16.
> +WAIT_FOR_STANDBY_CONFIRMATION "Waiting for the WAL to be received
> by
> physical standby."
>
> Moving the 'the' will make this more consistent with all other "Waiting for
> WAL..." names.
>
> SUGGESTION
> Waiting for WAL to be received by the physical standby.
Changed.
>
> ==========
> src/backend/utils/misc/guc_tables.c
>
> 17.
> + {
> + {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
> + gettext_noop("Lists streaming replication standby server slot "
> + "names that logical WAL sender processes will wait for."),
> + gettext_noop("Decoded changes are sent out to plugins by logical "
> + "WAL sender processes only after specified "
> + "replication slots confirm receiving WAL."), GUC_LIST_INPUT |
> + GUC_LIST_QUOTE }, &standby_slot_names, "", check_standby_slot_names,
> + assign_standby_slot_names, NULL },
>
> The wording of the detail msg feels kind of backwards to me.
>
> BEFORE
> Decoded changes are sent out to plugins by logical WAL sender processes
> only after specified replication slots confirm receiving WAL.
>
> SUGGESTION
> Logical WAL sender processes will send decoded changes to plugins only after
> the specified replication slots confirm receiving WAL.
Changed.
>
> ==========
> src/backend/utils/misc/postgresql.conf.sample
>
> 18.
> +#standby_slot_names = '' # streaming replication standby server slot
> +names that # logical walsender processes will wait for
>
> I'm not sure this is the best GUC name. See the general comment #0 above in
> this post.
As discussed, I didn’t change this.
>
> ==========
> src/include/replication/slot.h
>
> ==========
> src/include/replication/walsender.h
>
> ==========
> src/include/replication/walsender_private.h
>
> ==========
> src/include/utils/guc_hooks.h
>
> ==========
> src/test/recovery/t/006_logical_decoding.pl
>
> 19.
> +# Pass failover=true (last-arg), it should not have any impact on advancing.
>
> SUGGESTION
> Passing failover=true (last arg) should not have any impact on advancing.
Changed.
>
> ==========
> .../t/040_standby_failover_slots_sync.pl
>
> 20.
> +#
> +# | ----> standby1 (primary_slot_name = sb1_slot) # | ----> standby2
> +(primary_slot_name = sb2_slot) # primary ----- | # | ----> subscriber1
> +(failover = true) # | ----> subscriber2 (failover = false)
>
> In the diagram, the "--->" means a mixture of physical standbys and logical
> pub/sub replication. Maybe it can be a bit clearer?
>
> SUGGESTION
>
> # primary (publisher)
> #
> # (physical standbys)
> # | ----> standby1 (primary_slot_name = sb1_slot)
> # | ----> standby2 (primary_slot_name = sb2_slot)
> #
> # (logical replication)
> # | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
> # | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
>
I think one can distinguish it based on the 'standby' and 'subscriber' as well, because
'standby' normally refer to physical standby while the other refer to logical.
> ~~~
>
> 21.
> +# Set up is configured in such a way that the logical slot of
> +subscriber1 is # enabled failover, thus it will wait for the physical
> +slot of # standby1(sb1_slot) to catch up before sending decoded changes to
> subscriber1.
>
> /is enabled failover/is enabled for failover/
Changed.
>
> ~~~
>
> 22.
> +# Create another subscriber node without enabling failover, wait for
> +sync to # complete my $subscriber2 =
> +PostgreSQL::Test::Cluster->new('subscriber2');
> +$subscriber2->init;
> +$subscriber2->start;
> +$subscriber2->safe_psql(
> + 'postgres', qq[
> + CREATE TABLE tab_int (a int PRIMARY KEY); CREATE SUBSCRIPTION
> +regress_mysub2 CONNECTION '$publisher_connstr'
> PUBLICATION regress_mypub WITH (slot_name = lsub2_slot);
> +]);
> +
> +$subscriber1->wait_for_subscription_sync;
> +
>
> Is this meant to wait for 'subscription2'?
Yes, fixed.
>
> ~~~
>
> 23.
> # Stop the standby associated with the specified physical replication slot so #
> that the logical replication slot won't receive changes until the standby #
> comes up.
>
> Maybe this can give the values for better understanding:
>
> SUGGESTION
> Stop the standby associated with the specified physical replication slot
> (sb1_slot) so that the logical replication slot (lsub1_slot) won't receive changes
> until the standby comes up.
Changed.
>
> ~~~
>
> 24.
> +# Wait for the standby that's up and running gets the data from primary
>
> SUGGESTION
> Wait until the standby2 that's still running gets the data from the primary.
>
Changed.
> ~~~
>
> 25.
> +# Wait for the subscription that's up and running and is not enabled
> for failover.
> +# It gets the data from primary without waiting for any standbys.
>
> SUGGESTION
> Wait for subscription2 to get the data from the primary. This subscription was
> not enabled for failover so it gets the data without waiting for any standbys.
>
Changed.
> ~~~
>
> 26.
> +# The subscription that's up and running and is enabled for failover #
> +doesn't get the data from primary and keeps waiting for the # standby
> +specified in standby_slot_names.
>
> SUGGESTION
> The subscription1 was enabled for failover so it doesn't get the data from
> primary and keeps waiting for the standby specified in standby_slot_names
> (sb1_slot aka standby1).
>
Changed.
> ~~~
>
> 27.
> +# Start the standby specified in standby_slot_names and wait for it to
> +catch # up with the primary.
>
> SUGGESTION
> Start the standby specified in standby_slot_names (sb1_slot aka
> standby1) and wait for it to catch up with the primary.
>
Changed.
> ~~~
>
> 28.
> +# Now that the standby specified in standby_slot_names is up and
> +running, # primary must send the decoded changes to subscription
> +enabled for failover # While the standby was down, this subscriber
> +didn't receive any data from # primary i.e. the primary didn't allow it to go
> ahead of standby.
>
> SUGGESTION
> Now that the standby specified in standby_slot_names is up and running, the
> primary can send the decoded changes to the subscription enabled for failover
> (i.e. subscription1). While the standby was down,
> subscription1 didn't receive any data from the primary. i.e. the primary didn't
> allow it to go ahead of standby.
>
Changed.
> ~~~
>
> 29.
> +# Stop the standby associated with the specified physical replication
> +slot so # that the logical replication slot won't receive changes until
> +the standby # slot's restart_lsn is advanced or the slot is removed
> +from the # standby_slot_names list.
> +$primary->safe_psql('postgres', "TRUNCATE tab_int;");
> +$primary->wait_for_catchup('regress_mysub1');
> +$standby1->stop;
>
> Isn't this fragment more like the first step of the *next* TEST instead of the last
> step of this one?
>
Changed.
> ~~~
>
> 30.
> +##################################################
> +# Verify that when using pg_logical_slot_get_changes to consume changes
> +from a # logical slot with failover enabled, it will also wait for the
> +slots specified # in standby_slot_names to catch up.
> +##################################################
>
> AFAICT this test is checking only that the function cannot return while waiting
> for the stopped standby, but it doesn't seem to check that it *does* return
> when the stopped standby comes alive again.
>
Will think about this.
> ~~~
>
> 31.
> +$result =
> + $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM
> +tab_int;"); is($result, 't',
> + "subscriber1 doesn't get data as the sb1_slot doesn't catch up");
>
> Do you think this fragment should have a comment?
Added.
Attach the V100 patch set which addressed above comments.
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v100-0002-Document-the-steps-to-check-if-the-standby-is-r.patch (7.0K, ../../OS0PR01MB57160590C2CBED48976A113B94582@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v100-0002-Document-the-steps-to-check-if-the-standby-is-r.patch)
download | inline diff:
From 5ff64b5f7df334ac98a39d619d8fa9f473b55f3e Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Fri, 19 Jan 2024 11:04:16 +0530
Subject: [PATCH v100 2/2] Document the steps to check if the standby is ready
for failover
---
doc/src/sgml/high-availability.sgml | 9 ++
doc/src/sgml/logical-replication.sgml | 136 ++++++++++++++++++++++++++
2 files changed, 145 insertions(+)
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 236c0af65f..36215aa68c 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -1487,6 +1487,15 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)'
Written administration procedures are advised.
</para>
+ <para>
+ If you have opted for synchronization of logical slots (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>),
+ then before switching to the standby server, it is recommended to check
+ if the logical slots synchronized on the standby server are ready
+ for failover. This can be done by following the steps described in
+ <xref linkend="logical-replication-failover"/>.
+ </para>
+
<para>
To trigger failover of a log-shipping standby server, run
<command>pg_ctl promote</command> or call <function>pg_promote()</function>.
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ec2130669e..be59d306a1 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -687,6 +687,142 @@ ALTER SUBSCRIPTION
</sect1>
+ <sect1 id="logical-replication-failover">
+ <title>Logical Replication Failover</title>
+
+ <para>
+ When the publisher server is the primary server of a streaming replication,
+ the logical slots on that primary server can be synchronized to the standby
+ server by specifying <literal>failover = true</literal> when creating
+ subscriptions for those publications. Enabling failover ensures a seamless
+ transition of those subscriptions after the standby is promoted. They can
+ continue subscribing to publications now on the new primary server without
+ any data loss.
+ </para>
+
+ <para>
+ Because the slot synchronization logic copies asynchronously, it is
+ necessary to confirm that replication slots have been synced to the standby
+ server before the failover happens. Furthermore, to ensure a successful
+ failover, the standby server must not be lagging behind the subscriber. It
+ is highly recommended to use
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ to prevent the subscriber from consuming changes faster than the hot standby.
+ To confirm that the standby server is indeed ready for failover, follow
+ these 2 steps:
+ </para>
+
+ <procedure>
+ <step performance="required">
+ <para>
+ Confirm that all the necessary logical replication slots have been synced to
+ the standby server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node, use the following SQL to identify
+ which slots should be synced to the standby that we plan to promote.
+<programlisting>
+test_sub=# SELECT
+ array_agg(slotname) AS slots
+ FROM
+ ((
+ SELECT r.srsubid AS subid, CONCAT('pg_', srsubid, '_sync_', srrelid, '_', ctl.system_identifier) AS slotname
+ FROM pg_control_system() ctl, pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate = 'f' AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT s.oid AS subid, s.subslotname as slotname
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ slots
+-------
+ {sub1,sub2,sub3}
+(1 row)
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, check that the logical replication slots identified above exist on
+ the standby server and are ready for failover.
+<programlisting>
+test_standby=# SELECT slot_name, (synced AND NOT temporary AND conflict_reason IS NULL) AS failover_ready
+ FROM pg_replication_slots
+ WHERE slot_name IN ('sub1','sub2','sub3');
+ slot_name | failover_ready
+-------------+----------------
+ sub1 | t
+ sub2 | t
+ sub3 | t
+(3 rows)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+
+ <step performance="required">
+ <para>
+ Confirm that the standby server is not lagging behind the subscribers.
+ This step can be skipped if
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ has been correctly configured. If standby_slot_names is not configured
+ correctly, it is highly recommended to run this step after the primary
+ server is down, otherwise the results of the query may vary at different
+ points of time due to the ongoing replication on the logical subscribers
+ from the primary server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node check the last replayed WAL.
+ This step needs to be run on the database(s) that includes the failover
+ enabled subscription(s), to find the last replayed WAL on each database.
+<programlisting>
+test_sub=# SELECT
+ MAX(remote_lsn) AS remote_lsn_on_subscriber
+ FROM
+ ((
+ SELECT (CASE WHEN r.srsubstate = 'f' THEN pg_replication_origin_progress(CONCAT('pg_', r.srsubid, '_', r.srrelid), false)
+ WHEN r.srsubstate IN ('s', 'r') THEN r.srsublsn END) AS remote_lsn
+ FROM pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate IN ('f', 's', 'r') AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT pg_replication_origin_progress(CONCAT('pg_', s.oid), false) AS remote_lsn
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ remote_lsn_on_subscriber
+--------------------------
+ 0/3000388
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, on the standby server check that the last-received WAL location
+ is ahead of the replayed WAL location(s) on the subscriber identified
+ above. If the above SQL result was NULL, it means the subscriber has not
+ yet replayed any WAL, so the standby server must be ahead of the
+ subscriber, and this step can be skipped.
+<programlisting>
+test_standby=# SELECT pg_last_wal_receive_lsn() >= '0/3000388'::pg_lsn AS failover_ready;
+ failover_ready
+----------------
+ t
+(1 row)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+ </procedure>
+
+ <para>
+ If the result (<literal>failover_ready</literal>) of both above steps is
+ true, existing subscriptions will be able to continue without data loss.
+ </para>
+
+ </sect1>
+
<sect1 id="logical-replication-row-filter">
<title>Row Filters</title>
--
2.30.0.windows.2
[application/octet-stream] v100-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch (37.8K, ../../OS0PR01MB57160590C2CBED48976A113B94582@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v100-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch)
download | inline diff:
From 0f2640f30a5c0deb95b4f621492e1ea907a8bd00 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Fri, 23 Feb 2024 08:29:30 +0800
Subject: [PATCH v100] Allow logical walsenders to wait for the physical
standby
This patch introduces a mechanism to ensure that physical standby servers,
which are potential failover candidates, have received and flushed changes
before making them visible to subscribers. By doing so, it guarantees that
the promoted standby server is not lagging behind the subscribers when a
failover is necessary.
A new parameter named 'standby_slot_names' is introduced. The logical
walsender now guarantees that all local changes are sent and flushed to
the standby servers corresponding to the replication slots specified in
'standby_slot_names' before sending those changes to the subscriber.
Additionally, The SQL functions pg_logical_slot_get_changes and
pg_replication_slot_advance are modified to wait for the replication slots
mentioned in 'standby_slot_names' to catch up before returning.
---
doc/src/sgml/config.sgml | 29 ++
doc/src/sgml/logicaldecoding.sgml | 7 +
.../replication/logical/logicalfuncs.c | 12 +
src/backend/replication/logical/slotsync.c | 13 +
src/backend/replication/slot.c | 304 +++++++++++++++++-
src/backend/replication/slotfuncs.c | 12 +
src/backend/replication/walsender.c | 111 ++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 14 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/replication/slot.h | 6 +
src/include/replication/walsender.h | 1 +
src/include/replication/walsender_private.h | 7 +
src/include/utils/guc_hooks.h | 3 +
src/test/recovery/t/006_logical_decoding.pl | 3 +-
.../t/040_standby_failover_slots_sync.pl | 188 +++++++++++
16 files changed, 697 insertions(+), 16 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 36a2a5ce43..ce0fe0e9e1 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4420,6 +4420,35 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Lists the streaming replication standby server slot names that logical
+ WAL sender processes will wait for. Logical WAL sender processes will
+ send decoded changes to plugins only after the specified replication
+ slots confirm receiving WAL. This guarantees that logical replication
+ slots with failover enabled do not consume changes until those changes
+ are received and flushed to corresponding physical standbys. If a
+ logical replication connection is meant to switch to a physical standby
+ after the standby is promoted, the physical replication slot for the
+ standby should be listed here. Note that logical replication will not
+ proceed if the slots specified in the standby_slot_names do not exist or
+ are invalidated.
+ </para>
+ <para>
+ The standbys corresponding to the physical replication slots in
+ <varname>standby_slot_names</varname> must configure
+ <literal>sync_replication_slots = true</literal> so they can receive
+ failover logical slots changes from the primary.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 930c0fa8a6..5d16ea257d 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -384,6 +384,13 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU
must be enabled on the standby. It is also necessary to specify a valid
<literal>dbname</literal> in the
<link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>.
+ It's highly recommended that the said physical replication slot is named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ list on the primary, to prevent the subscriber from consuming changes
+ faster than the hot standby. Even when correctly configured, some latency
+ is expected when sending changes to logical subscribers due to the waiting
+ on slots named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>.
</para>
<para>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b0081d3ce5..f28d6c9015 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -109,6 +109,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
MemoryContext per_query_ctx;
MemoryContext oldcontext;
XLogRecPtr end_of_wal;
+ XLogRecPtr wait_for_wal_lsn;
LogicalDecodingContext *ctx;
ResourceOwner old_resowner = CurrentResourceOwner;
ArrayType *arr;
@@ -228,6 +229,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
NameStr(MyReplicationSlot->data.plugin),
format_procedure(fcinfo->flinfo->fn_oid))));
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to wait_for_wal_lsn.
+ */
+ if (XLogRecPtrIsInvalid(upto_lsn))
+ wait_for_wal_lsn = end_of_wal;
+ else
+ wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
+
+ WaitForStandbyConfirmation(wait_for_wal_lsn);
+
ctx->output_writer_private = p;
/*
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 36773cfe73..de05389d61 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -491,6 +491,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
latestFlushPtr = GetStandbyFlushRecPtr(NULL);
if (remote_slot->confirmed_lsn > latestFlushPtr)
{
+ /*
+ * Can get here only if GUC 'standby_slot_names' on the primary server
+ * was not configured correctly.
+ */
ereport(am_slotsync_worker ? LOG : ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("skipping slot synchronization as the received slot sync"
@@ -860,6 +864,15 @@ validate_remote_info(WalReceiverConn *wrconn)
remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull));
Assert(!isnull);
+ /*
+ * Slot sync is currently not supported on a cascading standby. This is
+ * because if we allow it, the primary server needs to wait for all the
+ * cascading standbys, otherwise, logical subscribers can still be ahead
+ * of one of the cascading standbys which we plan to promote. Thus, to
+ * avoid this additional complexity, we restrict it for the time being.
+ *
+ * XXX: If needed, this can be attempted in future.
+ */
if (remote_in_recovery)
ereport(ERROR,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 0f173f63a2..d711b352c8 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -46,13 +46,18 @@
#include "common/string.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/interrupt.h"
#include "replication/slotsync.h"
#include "replication/slot.h"
+#include "replication/walsender_private.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
+#include "utils/memutils.h"
+#include "utils/varlena.h"
/*
* Replication slot on-disk data structure.
@@ -115,10 +120,19 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
/* My backend's replication slot in the shared memory array */
ReplicationSlot *MyReplicationSlot = NULL;
-/* GUC variable */
+/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+/*
+ * This GUC lists streaming replication standby server slot names that
+ * logical WAL sender processes will wait for.
+ */
+char *standby_slot_names;
+
+/* This is parsed and cached list for raw standby_slot_names. */
+static List *standby_slot_names_list = NIL;
+
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
@@ -2345,3 +2359,291 @@ GetSlotInvalidationCause(const char *conflict_reason)
Assert(found);
return result;
}
+
+/*
+ * A helper function to validate slots specified in GUC standby_slot_names.
+ */
+static bool
+validate_standby_slots(char **newval)
+{
+ char *rawname;
+ List *elemlist;
+ bool ok;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Verify syntax and parse string into a list of identifiers */
+ ok = SplitIdentifierString(rawname, ',', &elemlist);
+
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
+
+ /*
+ * If the replication slots' data have been initialized, verify if the
+ * specified slots exist and are logical slots.
+ */
+ else if (ReplicationSlotCtl)
+ {
+ foreach_ptr(char, name, elemlist)
+ {
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, true);
+
+ if (!slot)
+ {
+ GUC_check_errdetail("replication slot \"%s\" does not exist",
+ name);
+ ok = false;
+ break;
+ }
+
+ if (!SlotIsPhysical(slot))
+ {
+ GUC_check_errdetail("\"%s\" is not a physical replication slot",
+ name);
+ ok = false;
+ break;
+ }
+ }
+ }
+
+ pfree(rawname);
+ list_free(elemlist);
+ return ok;
+}
+
+/*
+ * GUC check_hook for standby_slot_names
+ */
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ if ((*newval)[0] == '\0')
+ return true;
+
+ /*
+ * "*" is not accepted as in that case primary will not be able to know
+ * for which all standbys to wait for. Even if we have physical slots
+ * info, there is no way to confirm whether there is any standby
+ * configured for the known physical slots.
+ */
+ if (strcmp(*newval, "*") == 0)
+ {
+ GUC_check_errdetail("\"*\" is not accepted for standby_slot_names");
+ return false;
+ }
+
+ /* Now verify if the specified slots really exist and have correct type */
+ if (!validate_standby_slots(newval))
+ return false;
+
+ *extra = guc_strdup(ERROR, *newval);
+
+ return true;
+}
+
+/*
+ * GUC assign_hook for standby_slot_names
+ */
+void
+assign_standby_slot_names(const char *newval, void *extra)
+{
+ List *standby_slots;
+ MemoryContext oldcxt;
+ char *standby_slot_names_cpy = extra;
+
+ list_free(standby_slot_names_list);
+ standby_slot_names_list = NIL;
+
+ /* No value is specified for standby_slot_names. */
+ if (standby_slot_names_cpy == NULL)
+ return;
+
+ if (!SplitIdentifierString(standby_slot_names_cpy, ',', &standby_slots))
+ {
+ /* This should not happen if GUC checked check_standby_slot_names. */
+ elog(ERROR, "invalid list syntax");
+ }
+
+ /*
+ * Switch to the memory context under which GUC variables are allocated
+ * (GUCMemoryContext).
+ */
+ oldcxt = MemoryContextSwitchTo(GetMemoryChunkContext(standby_slot_names_cpy));
+ standby_slot_names_list = list_copy(standby_slots);
+ MemoryContextSwitchTo(oldcxt);
+}
+
+/*
+ * Return a copy of standby_slot_names_list if the copy flag is set to true,
+ * otherwise return the original list.
+ *
+ * Note that since we do not support syncing slots to cascading standbys, we
+ * return NIL if we are running in a standby to indicate that no standby slots
+ * need to be waited for.
+ */
+List *
+GetStandbySlotList(bool copy)
+{
+ if (RecoveryInProgress())
+ return NIL;
+
+ if (copy)
+ return list_copy(standby_slot_names_list);
+ else
+ return standby_slot_names_list;
+}
+
+/*
+ * Filter the standby slots based on the specified log sequence number
+ * (wait_for_lsn).
+ *
+ * This function updates the passed standby_slots list, removing any slots that
+ * have already caught up to or surpassed the given wait_for_lsn.
+ */
+void
+FilterStandbySlots(XLogRecPtr wait_for_lsn, List **standby_slots)
+{
+ if (*standby_slots == NIL)
+ return;
+
+ /*
+ * To prevent concurrent slot dropping and creation while filtering the
+ * slots, take the ReplicationSlotControlLock outside of the loop.
+ */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, *standby_slots)
+ {
+ XLogRecPtr restart_lsn;
+ bool invalidated;
+ bool inactive;
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ /*
+ * It may happen that the slot specified in standby_slot_names GUC
+ * value is dropped, so let's skip over it.
+ */
+ ereport(WARNING,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not exist",
+ name, "standby_slot_names"));
+ continue;
+ }
+
+ if (SlotIsLogical(slot))
+ {
+ /*
+ * If a logical slot name is provided in standby_slot_names, issue
+ * a WARNING and skip it. Although logical slots are disallowed in
+ * the GUC check_hook(validate_standby_slots), it is still
+ * possible for a user to drop an existing physical slot and
+ * recreate a logical slot with the same name.
+ */
+ ereport(WARNING,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot have logical replication slot \"%s\" in parameter %s",
+ name, "standby_slot_names"));
+ continue;
+ }
+
+ SpinLockAcquire(&slot->mutex);
+ restart_lsn = slot->data.restart_lsn;
+ invalidated = slot->data.invalidated != RS_INVAL_NONE;
+ inactive = slot->active_pid == 0;
+ SpinLockRelease(&slot->mutex);
+
+ if (invalidated)
+ {
+ /* Specified physical slot has been invalidated */
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("physical slot \"%s\" specified in parameter %s has been invalidated",
+ name, "standby_slot_names"));
+ continue;
+ }
+
+ if (XLogRecPtrIsInvalid(restart_lsn) || restart_lsn < wait_for_lsn)
+ {
+ /* Log warning if no active_pid for this physical slot */
+ if (inactive)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not have active_pid",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider starting standby associated with \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+
+ /* Continue if the current slot hasn't caught up. */
+ continue;
+ }
+
+ Assert(restart_lsn >= wait_for_lsn);
+ *standby_slots = foreach_delete_current(*standby_slots, name);
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+}
+
+/*
+ * Wait for physical standby to confirm receiving the given lsn.
+ *
+ * Used by logical decoding SQL functions that acquired failover enabled slot.
+ * It waits for physical standbys corresponding to the physical slots specified
+ * in the standby_slot_names GUC.
+ */
+void
+WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
+{
+ List *standby_slots;
+
+ if (!MyReplicationSlot->data.failover)
+ return;
+
+ standby_slots = GetStandbySlotList(true);
+
+ if (standby_slots == NIL)
+ return;
+
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+
+ for (;;)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ list_free(standby_slots);
+ standby_slots = GetStandbySlotList(true);
+ }
+
+ FilterStandbySlots(wait_for_lsn, &standby_slots);
+
+ /* Exit if done waiting for every slot. */
+ if (standby_slots == NIL)
+ break;
+
+ /*
+ * We wait for the slots in the standby_slot_names to catch up, but we
+ * use a timeout (1s) so we can also check if the standby_slot_names
+ * has been changed.
+ */
+ ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000,
+ WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION);
+ }
+
+ ConditionVariableCancelSleep();
+ list_free(standby_slots);
+}
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 768a304723..d864fe4133 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -464,6 +464,12 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* crash, but this makes the data consistent after a clean shutdown.
*/
ReplicationSlotMarkDirty();
+
+ /*
+ * Wake up logical walsenders holding failover enabled slots after
+ * updating the restart_lsn of the physical slot.
+ */
+ PhysicalWakeupLogicalWalSnd();
}
return retlsn;
@@ -504,6 +510,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto)
.segment_close = wal_segment_close),
NULL, NULL, NULL);
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to moveto lsn.
+ */
+ WaitForStandbyConfirmation(moveto);
+
/*
* Start reading at the slot's restart_lsn, which we know to point to
* a valid record.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 13bc3e0aee..aae19e3b5f 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1728,27 +1728,74 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
ProcessPendingWrites();
}
+/*
+ * Wake up the logical walsender processes with failover enabled slots if the
+ * currently acquired physical slot is specified in standby_slot_names GUC.
+ */
+void
+PhysicalWakeupLogicalWalSnd(void)
+{
+ List *standby_slots;
+
+ Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot));
+
+ standby_slots = GetStandbySlotList(false);
+
+ foreach_ptr(char, name, standby_slots)
+ {
+ if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0)
+ {
+ ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv);
+ return;
+ }
+ }
+}
+
/*
* Wait till WAL < loc is flushed to disk so it can be safely sent to client.
*
- * Returns end LSN of flushed WAL. Normally this will be >= loc, but
- * if we detect a shutdown request (either from postmaster or client)
- * we will return early, so caller must always check.
+ * If the walsender holds a logical slot that has enabled failover, we also
+ * wait for all the specified streaming replication standby servers to
+ * confirm receipt of WAL up to RecentFlushPtr.
+ *
+ * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we
+ * detect a shutdown request (either from postmaster or client) we will return
+ * early, so caller must always check.
*/
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
int wakeEvents;
+ bool wait_for_standby = false;
+ uint32 wait_event;
+ List *standby_slots = NIL;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
+ if (MyReplicationSlot->data.failover && replication_active)
+ standby_slots = GetStandbySlotList(true);
+
/*
- * Fast path to avoid acquiring the spinlock in case we already know we
- * have enough WAL available. This is particularly interesting if we're
- * far behind.
+ * Check if all the standby servers have confirmed receipt of WAL up to
+ * RecentFlushPtr even when we already know we have enough WAL available.
+ *
+ * Note that we cannot directly return without checking the status of
+ * standby servers because the standby_slot_names may have changed, which
+ * means there could be new standby slots in the list that have not yet
+ * caught up to the RecentFlushPtr.
*/
- if (RecentFlushPtr != InvalidXLogRecPtr &&
- loc <= RecentFlushPtr)
- return RecentFlushPtr;
+ if (!XLogRecPtrIsInvalid(RecentFlushPtr) && loc <= RecentFlushPtr)
+ {
+ FilterStandbySlots(RecentFlushPtr, &standby_slots);
+
+ /*
+ * Fast path to avoid acquiring the spinlock in case we already know
+ * we have enough WAL available and all the standby servers have
+ * confirmed receipt of WAL up to RecentFlushPtr. This is particularly
+ * interesting if we're far behind.
+ */
+ if (standby_slots == NIL)
+ return RecentFlushPtr;
+ }
/* Get a more recent flush pointer. */
if (!RecoveryInProgress())
@@ -1770,6 +1817,13 @@ WalSndWaitForWal(XLogRecPtr loc)
{
ConfigReloadPending = false;
ProcessConfigFile(PGC_SIGHUP);
+
+ if (MyReplicationSlot->data.failover && replication_active)
+ {
+ list_free(standby_slots);
+ standby_slots = GetStandbySlotList(true);
+ }
+
SyncRepInitConfig();
}
@@ -1784,8 +1838,18 @@ WalSndWaitForWal(XLogRecPtr loc)
if (got_STOPPING)
XLogBackgroundFlush();
+ /*
+ * Update the standby slots that have not yet caught up to the flushed
+ * position. It is good to wait up to RecentFlushPtr and then let it
+ * send the changes to logical subscribers one by one which are
+ * already covered in RecentFlushPtr without needing to wait on every
+ * change for standby confirmation.
+ */
+ if (wait_for_standby)
+ FilterStandbySlots(RecentFlushPtr, &standby_slots);
+
/* Update our idea of the currently flushed position. */
- if (!RecoveryInProgress())
+ else if (!RecoveryInProgress())
RecentFlushPtr = GetFlushRecPtr(NULL);
else
RecentFlushPtr = GetXLogReplayRecPtr(NULL);
@@ -1813,9 +1877,18 @@ WalSndWaitForWal(XLogRecPtr loc)
!waiting_for_ping_response)
WalSndKeepalive(false, InvalidXLogRecPtr);
- /* check whether we're done */
- if (loc <= RecentFlushPtr)
+ if (loc > RecentFlushPtr)
+ wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
+ else if (standby_slots)
+ {
+ wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
+ wait_for_standby = true;
+ }
+ else
+ {
+ /* Already caught up and doesn't need to wait for standby_slots. */
break;
+ }
/* Waiting for new WAL. Since we need to wait, we're now caught up. */
WalSndCaughtUp = true;
@@ -1855,9 +1928,11 @@ WalSndWaitForWal(XLogRecPtr loc)
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL);
+ WalSndWait(wakeEvents, sleeptime, wait_event);
}
+ list_free(standby_slots);
+
/* reactivate latch so WalSndLoop knows to continue */
SetLatch(MyLatch);
return RecentFlushPtr;
@@ -2265,6 +2340,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn)
{
ReplicationSlotMarkDirty();
ReplicationSlotsComputeRequiredLSN();
+ PhysicalWakeupLogicalWalSnd();
}
/*
@@ -3538,6 +3614,7 @@ WalSndShmemInit(void)
ConditionVariableInit(&WalSndCtl->wal_flush_cv);
ConditionVariableInit(&WalSndCtl->wal_replay_cv);
+ ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv);
}
}
@@ -3607,8 +3684,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event)
*
* And, we use separate shared memory CVs for physical and logical
* walsenders for selective wake ups, see WalSndWakeup() for more details.
+ *
+ * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV
+ * until awakened by physical walsenders after the walreceiver confirms
+ * the receipt of the LSN.
*/
- if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
+ if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+ else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv);
else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 4fffb46625..01fe180b65 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP
LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server."
LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server."
SSL_OPEN_SERVER "Waiting for SSL while attempting connection."
+WAIT_FOR_STANDBY_CONFIRMATION "Waiting for WAL to be received and flushed by the physical standby."
WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process."
WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 527a2b2734..6ee6375cb8 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4639,6 +4639,20 @@ struct config_string ConfigureNamesString[] =
check_debug_io_direct, assign_debug_io_direct, NULL
},
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("Lists streaming replication standby server slot "
+ "names that logical WAL sender processes will wait for."),
+ gettext_noop("Logical WAL sender processes will send decoded "
+ "changes to plugins only after the specified "
+ "replication slots confirm receiving WAL."),
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, assign_standby_slot_names, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c97f9a25f0..cadfe10958 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -334,6 +334,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsender processes will wait for
# - Standby Servers -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index acbf567150..04a2717138 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -226,6 +226,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
+extern PGDLLIMPORT char *standby_slot_names;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -274,4 +275,9 @@ extern void CheckSlotPermissions(void);
extern ReplicationSlotInvalidationCause
GetSlotInvalidationCause(const char *conflict_reason);
+extern List *GetStandbySlotList(bool copy);
+extern void FilterStandbySlots(XLogRecPtr wait_for_lsn,
+ List **standby_slots);
+extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn);
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index 0c3996e926..f2d8297f01 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -39,6 +39,7 @@ extern void InitWalSender(void);
extern bool exec_replication_command(const char *cmd_string);
extern void WalSndErrorCleanup(void);
extern void WalSndResourceCleanup(bool isCommit);
+extern void PhysicalWakeupLogicalWalSnd(void);
extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
extern void WalSndSignals(void);
extern Size WalSndShmemSize(void);
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 3113e9ea47..3c134c8edf 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -113,6 +113,13 @@ typedef struct
ConditionVariable wal_flush_cv;
ConditionVariable wal_replay_cv;
+ /*
+ * Used by physical walsenders holding slots specified in
+ * standby_slot_names to wake up logical walsenders holding failover
+ * enabled slots when a walreceiver confirms the receipt of LSN.
+ */
+ ConditionVariable wal_confirm_rcv_cv;
+
WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER];
} WalSndCtlData;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 339c490300..dcf9a040d1 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -163,5 +163,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra,
extern void assign_wal_consistency_checking(const char *newval, void *extra);
extern bool check_wal_segment_size(int *newval, void **extra, GucSource source);
extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
+extern bool check_standby_slot_names(char **newval, void **extra,
+ GucSource source);
+extern void assign_standby_slot_names(const char *newval, void *extra);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl
index 5c7b4ca5e3..b95d95c06f 100644
--- a/src/test/recovery/t/006_logical_decoding.pl
+++ b/src/test/recovery/t/006_logical_decoding.pl
@@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'},
undef, 'logical slot was actually dropped with DB');
# Test logical slot advancing and its durability.
+# Passing failover=true (last arg) should not have any impact on advancing.
my $logical_slot = 'logical_slot';
$node_primary->safe_psql('postgres',
- "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);"
+ "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);"
);
$node_primary->psql(
'postgres', "
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 968aa7b05b..826c68b5bb 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -446,11 +446,199 @@ ok( $standby1->poll_query_until(
"SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot' AND synced AND NOT temporary;"),
'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby');
+##################################################
+# Test primary disallowing specified logical replication slots getting ahead of
+# specified physical replication slots. It uses the following set up:
+#
+# | ----> standby1 (primary_slot_name = sb1_slot)
+# | ----> standby2 (primary_slot_name = sb2_slot)
+# primary ----- |
+# | ----> subscriber1 (failover = true)
+# | ----> subscriber2 (failover = false)
+#
+# standby_slot_names = 'sb1_slot'
+#
+# Set up is configured in such a way that the logical slot of subscriber1 is
+# enabled for failover, thus it will wait for the physical slot of
+# standby1(sb1_slot) to catch up before sending decoded changes to subscriber1.
+##################################################
+
+$backup_name = 'backup3';
+
+$primary->psql('postgres',
+ q{SELECT pg_create_physical_replication_slot('sb2_slot');});
+
+$primary->backup($backup_name);
+
+# Create another standby
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup(
+ $primary, $backup_name,
+ has_streaming => 1,
+ has_restoring => 1);
+$standby2->append_conf(
+ 'postgresql.conf', qq(
+primary_slot_name = 'sb2_slot'
+));
+$standby2->start;
+$primary->wait_for_replay_catchup($standby2);
+
+# Configure primary to disallow any logical slots that enabled failover from
+# getting ahead of specified physical replication slot (sb1_slot).
+$primary->append_conf(
+ 'postgresql.conf', qq(
+standby_slot_names = 'sb1_slot'
+));
+$primary->reload;
+
+# Create another subscriber node without enabling failover, wait for sync to
+# complete
+my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$subscriber2->init;
+$subscriber2->start;
+$subscriber2->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE tab_int (a int PRIMARY KEY);
+ CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot);
+]);
+
+$subscriber2->wait_for_subscription_sync;
+
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+# Stop the standby associated with the specified physical replication slot
+# (sb1_slot) so that the logical replication slot (lsub1_slot) won't receive
+# changes until the standby comes up.
+$standby1->stop;
+
+# Create some data on the primary
+my $primary_row_count = 20;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(11, $primary_row_count);");
+
+# Wait until the standby2 that's still running gets the data from the primary
+$primary->wait_for_replay_catchup($standby2);
+$result = $standby2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby2 gets data from primary");
+
+# Wait for regress_mysub2 to get the data from the primary. This subscription
+# was not enabled for failover so it gets the data without waiting for any
+# standbys.
+$primary->wait_for_catchup('regress_mysub2');
+$result = $subscriber2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "subscriber2 gets data from primary");
+
+# The regress_mysub1 was enabled for failover so it doesn't get the data from
+# primary and keeps waiting for the standby specified in standby_slot_names
+# (sb1_slot aka standby1).
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) <> $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data from primary until standby1 acknowledges changes"
+);
+
+# Start the standby specified in standby_slot_names (sb1_slot aka standby1) and
+# wait for it to catch up with the primary.
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+$result = $standby1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby1 gets data from primary");
+
+# Now that the standby specified in standby_slot_names is up and running, the
+# primary can send the decoded changes to the subscription enabled for failover
+# (i.e. regress_mysub1). While the standby was down, regress_mysub1 didn't
+# receive any data from the primary. i.e. the primary didn't allow it to go
+# ahead of standby.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 acknowledges changes");
+
+##################################################
+# Verify that when using pg_logical_slot_get_changes to consume changes from a
+# logical slot with failover enabled, it will also wait for the slots specified
+# in standby_slot_names to catch up.
+##################################################
+
+# Stop the standby associated with the specified physical replication slot so
+# that the logical replication slot won't receive changes until the standby
+# slot's restart_lsn is advanced or the slot is removed from the
+# standby_slot_names list.
+$primary->safe_psql('postgres', "TRUNCATE tab_int;");
+$primary->wait_for_catchup('regress_mysub1');
+$standby1->stop;
+
+# Create a logical 'test_decoding' replication slot with failover enabled
+$primary->safe_psql('postgres',
+ "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);"
+);
+
+my $back_q = $primary->background_psql('postgres', on_error_stop => 0);
+my $pid = $back_q->query('SELECT pg_backend_pid()');
+
+# Try and get changes from the logical slot with failover enabled.
+my $offset = -s $primary->logfile;
+$back_q->query_until(qr//,
+ "SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);\n");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+ok($primary->safe_psql('postgres', "SELECT pg_cancel_backend($pid)"),
+ "cancelling pg_logical_slot_get_changes command");
+
+$back_q->quit;
+
+$primary->safe_psql('postgres',
+ "SELECT pg_drop_replication_slot('test_slot');"
+);
+
+##################################################
+# Test that logical replication will wait for the user-created inactive
+# physical slot to catch up until we remove the slot from standby_slot_names.
+##################################################
+
+# Create some data on the primary
+$primary_row_count = 10;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);");
+
+# The regress_mysub1 doesn't get the data from primary because the specified
+# standby slot (sb1_slot) in standby_slot_names is inactive.
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data as the sb1_slot doesn't catch up");
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the primary server should now
+# send the decoded changes to the subscription.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list"
+);
+
##################################################
# Promote the standby1 to primary. Confirm that:
# a) the slot 'lsub1_slot' is retained on the new primary
# b) logical replication for regress_mysub1 is resumed successfully after failover
##################################################
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+
$standby1->promote;
# Update subscription with the new primary's connection info
--
2.30.0.windows.2
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-02-28 11:35 Bertrand Drouvot <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
1 sibling, 1 reply; 68+ messages in thread
From: Bertrand Drouvot @ 2024-02-28 11:35 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
Hi,
On Wed, Feb 28, 2024 at 02:23:27AM +0000, Zhijie Hou (Fujitsu) wrote:
> Attach the V100 patch set which addressed above comments.
Thanks!
A few random comments:
1 ===
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
What about to get rid of the brackets here?
2 ===
+
+ /*
+ * If the replication slots' data have been initialized, verify if the
+ * specified slots exist and are logical slots.
+ */
remove the empty line above the comment?
3 ===
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ if ((*newval)[0] == '\0')
+ return true;
I think "**newval == '\0'" is easier to read but that's a matter of taste and
check_synchronous_standby_names() is already using the same so it's a nit.
4 ===
Regarding the test, what about adding one to test the "new" behavior discussed
up-thread? (logical replication will wait if slot mentioned in standby_slot_names
is dropped and/or does not exist when the engine starts?)
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-02-29 02:59 Peter Smith <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
1 sibling, 1 reply; 68+ messages in thread
From: Peter Smith @ 2024-02-29 02:59 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Wed, Feb 28, 2024 at 1:23 PM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> On Tuesday, February 27, 2024 3:18 PM Peter Smith <[email protected]> wrote:
...
> > 20.
> > +#
> > +# | ----> standby1 (primary_slot_name = sb1_slot) # | ----> standby2
> > +(primary_slot_name = sb2_slot) # primary ----- | # | ----> subscriber1
> > +(failover = true) # | ----> subscriber2 (failover = false)
> >
> > In the diagram, the "--->" means a mixture of physical standbys and logical
> > pub/sub replication. Maybe it can be a bit clearer?
> >
> > SUGGESTION
> >
> > # primary (publisher)
> > #
> > # (physical standbys)
> > # | ----> standby1 (primary_slot_name = sb1_slot)
> > # | ----> standby2 (primary_slot_name = sb2_slot)
> > #
> > # (logical replication)
> > # | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
> > # | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
> >
>
> I think one can distinguish it based on the 'standby' and 'subscriber' as well, because
> 'standby' normally refer to physical standby while the other refer to logical.
>
Ok, but shouldn't it at least include info about the logical slot
names associated with the subscribers (slot_name = lsub1_slot,
slot_name = lsub2_slot) like suggested above?
======
Here are some more review comments for v100-0001
======
doc/src/sgml/config.sgml
1.
+ <para>
+ Lists the streaming replication standby server slot names that logical
+ WAL sender processes will wait for. Logical WAL sender processes will
+ send decoded changes to plugins only after the specified replication
+ slots confirm receiving WAL. This guarantees that logical replication
+ slots with failover enabled do not consume changes until those changes
+ are received and flushed to corresponding physical standbys. If a
+ logical replication connection is meant to switch to a physical standby
+ after the standby is promoted, the physical replication slot for the
+ standby should be listed here. Note that logical replication will not
+ proceed if the slots specified in the standby_slot_names do
not exist or
+ are invalidated.
+ </para>
Is that note ("Note that logical replication will not proceed if the
slots specified in the standby_slot_names do not exist or are
invalidated") meant only for subscriptions marked for 'failover' or
any subscription? Maybe wording can be modified to help clarify it?
======
src/backend/replication/slot.c
2.
+/*
+ * A helper function to validate slots specified in GUC standby_slot_names.
+ */
+static bool
+validate_standby_slots(char **newval)
+{
+ char *rawname;
+ List *elemlist;
+ bool ok;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Verify syntax and parse string into a list of identifiers */
+ ok = SplitIdentifierString(rawname, ',', &elemlist);
+
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
+
+ /*
+ * If the replication slots' data have been initialized, verify if the
+ * specified slots exist and are logical slots.
+ */
+ else if (ReplicationSlotCtl)
+ {
+ foreach_ptr(char, name, elemlist)
+ {
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, true);
+
+ if (!slot)
+ {
+ GUC_check_errdetail("replication slot \"%s\" does not exist",
+ name);
+ ok = false;
+ break;
+ }
+
+ if (!SlotIsPhysical(slot))
+ {
+ GUC_check_errdetail("\"%s\" is not a physical replication slot",
+ name);
+ ok = false;
+ break;
+ }
+ }
+ }
+
+ pfree(rawname);
+ list_free(elemlist);
+ return ok;
+}
2a.
I didn't mention this previously because I thought this function was
not going to change anymore, but since Bertrand suggested some changes
[1], I will say IMO the { } are fine here for the single statement,
but I think it will be better to rearrange this code to be like below.
Having a 2nd NOP 'else' gives a much better place where you can put
your ReplicationSlotCtl comment.
if (!ok)
{
GUC_check_errdetail("List syntax is invalid.");
}
else if (!ReplicationSlotCtl)
{
<Insert big comment here about why it is OK to skip when
ReplicationSlotCtl is NULL>
}
else
{
foreach_ptr ...
}
~
2b.
In any case even if don't refactor anything I still think you need to
extend that comment to explain how skipping ReplicationSlotCtl is NULL
is only OK because there will be some later checking also done in the
FilterStandbySlots() function to catch any config problems.
------
[1] https://www.postgresql.org/message-id/Zd8ahZXw82ieFxX/%40ip-10-97-1-34.eu-west-3.compute.internal
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-02-29 03:16 Zhijie Hou (Fujitsu) <[email protected]>
parent: Bertrand Drouvot <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-02-29 03:16 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Wednesday, February 28, 2024 7:36 PM Bertrand Drouvot <[email protected]> wrote:
>
> On Wed, Feb 28, 2024 at 02:23:27AM +0000, Zhijie Hou (Fujitsu) wrote:
> > Attach the V100 patch set which addressed above comments.
>
> A few random comments:
Thanks for the comments!
>
> 1 ===
>
> + if (!ok)
> + {
> + GUC_check_errdetail("List syntax is invalid.");
> + }
>
> What about to get rid of the brackets here?
I personally prefer the current style.
>
> 2 ===
>
> +
> + /*
> + * If the replication slots' data have been initialized, verify if the
> + * specified slots exist and are logical slots.
> + */
>
> remove the empty line above the comment?
I feel it would be clean to have an empty line before the comments.
>
> 3 ===
>
> +check_standby_slot_names(char **newval, void **extra, GucSource source)
> +{
> + if ((*newval)[0] == '\0')
> + return true;
>
> I think "**newval == '\0'" is easier to read but that's a matter of taste and
> check_synchronous_standby_names() is already using the same so it's a nit.
I don't have a strong opinion on this, so will change if others also feel so.
>
> 4 ===
>
> Regarding the test, what about adding one to test the "new" behavior
> discussed up-thread? (logical replication will wait if slot mentioned in
> standby_slot_names is dropped and/or does not exist when the engine starts?)
Will think about this.
Best Regards,
Hou zj
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-02-29 03:43 Peter Smith <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Peter Smith @ 2024-02-29 03:43 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Tue, Feb 27, 2024 at 11:35 PM Amit Kapila <[email protected]> wrote:
>
> On Tue, Feb 27, 2024 at 12:48 PM Peter Smith <[email protected]> wrote:
> >
> > Here are some review comments for v99-0001
> >
> > ==========
> > 0. GENERAL.
> >
> > +#standby_slot_names = '' # streaming replication standby server slot names that
> > + # logical walsender processes will wait for
> >
> > IMO the GUC name is too generic. There is nothing in this name to
> > suggest it has anything to do with logical slot synchronization; that
> > meaning is only found in the accompanying comment -- it would be
> > better if the GUC name itself were more self-explanatory.
> >
> > e.g. Maybe like 'wal_sender_sync_standby_slot_names' or
> > 'wal_sender_standby_slot_names', 'wal_sender_wait_for_standby_slots',
> > or ...
> >
>
> It would be wrong and or misleading to append wal_sender to this GUC
> name as this is used during SQL APIs as well.
Fair enough, but the fact that some SQL functions might wait is also
not mentioned in the config file comment, nor in the documentation for
GUC 'standby_slot_names'. Seems like a docs omission?
> Also, adding wait sounds
> more like a boolean. So, I don't see the proposed names any better
> than the current one.
>
Anyway, the point is that the current GUC name 'standby_slot_names' is
not ideal IMO because it doesn't have enough meaning by itself -- e.g.
you have to read the accompanying comment or documentation to have any
idea of its purpose.
My suggested GUC names were mostly just to get people thinking about
it. Maybe others can come up with better names.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-02-29 09:53 Amit Kapila <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 2 replies; 68+ messages in thread
From: Amit Kapila @ 2024-02-29 09:53 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Thu, Feb 29, 2024 at 8:29 AM Peter Smith <[email protected]> wrote:
>
> On Wed, Feb 28, 2024 at 1:23 PM Zhijie Hou (Fujitsu)
> <[email protected]> wrote:
> >
> > On Tuesday, February 27, 2024 3:18 PM Peter Smith <[email protected]> wrote:
> ...
> > > 20.
> > > +#
> > > +# | ----> standby1 (primary_slot_name = sb1_slot) # | ----> standby2
> > > +(primary_slot_name = sb2_slot) # primary ----- | # | ----> subscriber1
> > > +(failover = true) # | ----> subscriber2 (failover = false)
> > >
> > > In the diagram, the "--->" means a mixture of physical standbys and logical
> > > pub/sub replication. Maybe it can be a bit clearer?
> > >
> > > SUGGESTION
> > >
> > > # primary (publisher)
> > > #
> > > # (physical standbys)
> > > # | ----> standby1 (primary_slot_name = sb1_slot)
> > > # | ----> standby2 (primary_slot_name = sb2_slot)
> > > #
> > > # (logical replication)
> > > # | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
> > > # | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
> > >
> >
> > I think one can distinguish it based on the 'standby' and 'subscriber' as well, because
> > 'standby' normally refer to physical standby while the other refer to logical.
> >
I think Peter's suggestion will make the setup clear.
>
> Ok, but shouldn't it at least include info about the logical slot
> names associated with the subscribers (slot_name = lsub1_slot,
> slot_name = lsub2_slot) like suggested above?
>
> ======
>
> Here are some more review comments for v100-0001
>
> ======
> doc/src/sgml/config.sgml
>
> 1.
> + <para>
> + Lists the streaming replication standby server slot names that logical
> + WAL sender processes will wait for. Logical WAL sender processes will
> + send decoded changes to plugins only after the specified replication
> + slots confirm receiving WAL. This guarantees that logical replication
> + slots with failover enabled do not consume changes until those changes
> + are received and flushed to corresponding physical standbys. If a
> + logical replication connection is meant to switch to a physical standby
> + after the standby is promoted, the physical replication slot for the
> + standby should be listed here. Note that logical replication will not
> + proceed if the slots specified in the standby_slot_names do
> not exist or
> + are invalidated.
> + </para>
>
> Is that note ("Note that logical replication will not proceed if the
> slots specified in the standby_slot_names do not exist or are
> invalidated") meant only for subscriptions marked for 'failover' or
> any subscription? Maybe wording can be modified to help clarify it?
>
I think it is implicit that here we are talking about failover slots.
I think clarifying again the same could be repetitive considering the
previous sentence: "This guarantees that logical replication slots
with failover enabled do not consume .." have mentioned it.
> ======
> src/backend/replication/slot.c
>
> 2.
> +/*
> + * A helper function to validate slots specified in GUC standby_slot_names.
> + */
> +static bool
> +validate_standby_slots(char **newval)
> +{
> + char *rawname;
> + List *elemlist;
> + bool ok;
> +
> + /* Need a modifiable copy of string */
> + rawname = pstrdup(*newval);
> +
> + /* Verify syntax and parse string into a list of identifiers */
> + ok = SplitIdentifierString(rawname, ',', &elemlist);
> +
> + if (!ok)
> + {
> + GUC_check_errdetail("List syntax is invalid.");
> + }
> +
> + /*
> + * If the replication slots' data have been initialized, verify if the
> + * specified slots exist and are logical slots.
> + */
> + else if (ReplicationSlotCtl)
> + {
> + foreach_ptr(char, name, elemlist)
> + {
> + ReplicationSlot *slot;
> +
> + slot = SearchNamedReplicationSlot(name, true);
> +
> + if (!slot)
> + {
> + GUC_check_errdetail("replication slot \"%s\" does not exist",
> + name);
> + ok = false;
> + break;
> + }
> +
> + if (!SlotIsPhysical(slot))
> + {
> + GUC_check_errdetail("\"%s\" is not a physical replication slot",
> + name);
> + ok = false;
> + break;
> + }
> + }
> + }
> +
> + pfree(rawname);
> + list_free(elemlist);
> + return ok;
> +}
>
> 2a.
> I didn't mention this previously because I thought this function was
> not going to change anymore, but since Bertrand suggested some changes
> [1], I will say IMO the { } are fine here for the single statement,
> but I think it will be better to rearrange this code to be like below.
> Having a 2nd NOP 'else' gives a much better place where you can put
> your ReplicationSlotCtl comment.
>
> if (!ok)
> {
> GUC_check_errdetail("List syntax is invalid.");
> }
> else if (!ReplicationSlotCtl)
> {
> <Insert big comment here about why it is OK to skip when
> ReplicationSlotCtl is NULL>
> }
> else
> {
> foreach_ptr ...
> }
>
+1. This will make the code and reasoning to skip clear.
Few additional comments on the latest patch:
=================================
1.
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
...
+ if (!XLogRecPtrIsInvalid(RecentFlushPtr) && loc <= RecentFlushPtr &&
+ (!replication_active || StandbyConfirmedFlush(loc, WARNING)))
+ {
+ /*
+ * Fast path to avoid acquiring the spinlock in case we already know
+ * we have enough WAL available and all the standby servers have
+ * confirmed receipt of WAL up to RecentFlushPtr. This is particularly
+ * interesting if we're far behind.
+ */
return RecentFlushPtr;
+ }
...
...
+ * Wait for WALs to be flushed to disk only if the postmaster has not
+ * asked us to stop.
+ */
+ if (loc > RecentFlushPtr && !got_STOPPING)
+ wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
+
+ /*
+ * Check if the standby slots have caught up to the flushed position.
+ * It is good to wait up to RecentFlushPtr and then let it send the
+ * changes to logical subscribers one by one which are already covered
+ * in RecentFlushPtr without needing to wait on every change for
+ * standby confirmation. Note that after receiving the shutdown signal,
+ * an ERROR is reported if any slots are dropped, invalidated, or
+ * inactive. This measure is taken to prevent the walsender from
+ * waiting indefinitely.
+ */
+ else if (replication_active &&
+ !StandbyConfirmedFlush(RecentFlushPtr, WARNING))
+ {
+ wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
+ wait_for_standby = true;
+ }
+ else
+ {
+ /* Already caught up and doesn't need to wait for standby_slots. */
break;
+ }
...
}
Can we try to move these checks into a separate function that returns
a boolean and has an out parameter as wait_event?
2. How about naming StandbyConfirmedFlush() as StandbySlotsAreCaughtup?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-02-29 10:08 Amit Kapila <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Amit Kapila @ 2024-02-29 10:08 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Thu, Feb 29, 2024 at 9:13 AM Peter Smith <[email protected]> wrote:
>
> On Tue, Feb 27, 2024 at 11:35 PM Amit Kapila <[email protected]> wrote:
> >
>
> > Also, adding wait sounds
> > more like a boolean. So, I don't see the proposed names any better
> > than the current one.
> >
>
> Anyway, the point is that the current GUC name 'standby_slot_names' is
> not ideal IMO because it doesn't have enough meaning by itself -- e.g.
> you have to read the accompanying comment or documentation to have any
> idea of its purpose.
>
Yeah, one has to read the description but that is true for other
parameters like "temp_tablespaces". I don't have any better ideas but
open to suggestions.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-02-29 11:17 Amit Kapila <[email protected]>
parent: Amit Kapila <[email protected]>
1 sibling, 1 reply; 68+ messages in thread
From: Amit Kapila @ 2024-02-29 11:17 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Thu, Feb 29, 2024 at 3:23 PM Amit Kapila <[email protected]> wrote:
>
> Few additional comments on the latest patch:
> =================================
> 1.
> static XLogRecPtr
> WalSndWaitForWal(XLogRecPtr loc)
> {
> ...
> + if (!XLogRecPtrIsInvalid(RecentFlushPtr) && loc <= RecentFlushPtr &&
> + (!replication_active || StandbyConfirmedFlush(loc, WARNING)))
> + {
> + /*
> + * Fast path to avoid acquiring the spinlock in case we already know
> + * we have enough WAL available and all the standby servers have
> + * confirmed receipt of WAL up to RecentFlushPtr. This is particularly
> + * interesting if we're far behind.
> + */
> return RecentFlushPtr;
> + }
> ...
> ...
> + * Wait for WALs to be flushed to disk only if the postmaster has not
> + * asked us to stop.
> + */
> + if (loc > RecentFlushPtr && !got_STOPPING)
> + wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
> +
> + /*
> + * Check if the standby slots have caught up to the flushed position.
> + * It is good to wait up to RecentFlushPtr and then let it send the
> + * changes to logical subscribers one by one which are already covered
> + * in RecentFlushPtr without needing to wait on every change for
> + * standby confirmation. Note that after receiving the shutdown signal,
> + * an ERROR is reported if any slots are dropped, invalidated, or
> + * inactive. This measure is taken to prevent the walsender from
> + * waiting indefinitely.
> + */
> + else if (replication_active &&
> + !StandbyConfirmedFlush(RecentFlushPtr, WARNING))
> + {
> + wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
> + wait_for_standby = true;
> + }
> + else
> + {
> + /* Already caught up and doesn't need to wait for standby_slots. */
> break;
> + }
> ...
> }
>
> Can we try to move these checks into a separate function that returns
> a boolean and has an out parameter as wait_event?
>
> 2. How about naming StandbyConfirmedFlush() as StandbySlotsAreCaughtup?
>
Some more comments:
==================
1.
+ foreach_ptr(char, name, elemlist)
+ {
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, true);
+
+ if (!slot)
+ {
+ GUC_check_errdetail("replication slot \"%s\" does not exist",
+ name);
+ ok = false;
+ break;
+ }
+
+ if (!SlotIsPhysical(slot))
+ {
+ GUC_check_errdetail("\"%s\" is not a physical replication slot",
+ name);
+ ok = false;
+ break;
+ }
+ }
Won't the second check need protection via ReplicationSlotControlLock?
2. In WaitForStandbyConfirmation(), we are anyway calling
StandbyConfirmedFlush() before the actual sleep in the loop, so does
calling it at the beginning of the function will serve any purpose? If
so, it is better to add some comments explaining the same.
3. Also do we need to perform the validation checks done in
StandbyConfirmedFlush() repeatedly when it is invoked in a loop? We
can probably separate those checks and perform them just once.
4.
+ *
+ * XXX: If needed, this can be attempted in future.
Remove this part of the comment.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-01 02:17 Zhijie Hou (Fujitsu) <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 2 replies; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-01 02:17 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Thursday, February 29, 2024 7:17 PM Amit Kapila <[email protected]> wrote:
>
> On Thu, Feb 29, 2024 at 3:23 PM Amit Kapila <[email protected]>
> wrote:
> >
> > Few additional comments on the latest patch:
> > =================================
> > 1.
> > static XLogRecPtr
> > WalSndWaitForWal(XLogRecPtr loc)
> > {
> > ...
> > + if (!XLogRecPtrIsInvalid(RecentFlushPtr) && loc <= RecentFlushPtr &&
> > + (!replication_active || StandbyConfirmedFlush(loc, WARNING))) {
> > + /*
> > + * Fast path to avoid acquiring the spinlock in case we already know
> > + * we have enough WAL available and all the standby servers have
> > + * confirmed receipt of WAL up to RecentFlushPtr. This is
> > + particularly
> > + * interesting if we're far behind.
> > + */
> > return RecentFlushPtr;
> > + }
> > ...
> > ...
> > + * Wait for WALs to be flushed to disk only if the postmaster has not
> > + * asked us to stop.
> > + */
> > + if (loc > RecentFlushPtr && !got_STOPPING) wait_event =
> > + WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
> > +
> > + /*
> > + * Check if the standby slots have caught up to the flushed position.
> > + * It is good to wait up to RecentFlushPtr and then let it send the
> > + * changes to logical subscribers one by one which are already
> > + covered
> > + * in RecentFlushPtr without needing to wait on every change for
> > + * standby confirmation. Note that after receiving the shutdown
> > + signal,
> > + * an ERROR is reported if any slots are dropped, invalidated, or
> > + * inactive. This measure is taken to prevent the walsender from
> > + * waiting indefinitely.
> > + */
> > + else if (replication_active &&
> > + !StandbyConfirmedFlush(RecentFlushPtr, WARNING)) { wait_event =
> > + WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
> > + wait_for_standby = true;
> > + }
> > + else
> > + {
> > + /* Already caught up and doesn't need to wait for standby_slots. */
> > break;
> > + }
> > ...
> > }
> >
> > Can we try to move these checks into a separate function that returns
> > a boolean and has an out parameter as wait_event?
> >
> > 2. How about naming StandbyConfirmedFlush() as
> StandbySlotsAreCaughtup?
> >
>
> Some more comments:
> ==================
> 1.
> + foreach_ptr(char, name, elemlist)
> + {
> + ReplicationSlot *slot;
> +
> + slot = SearchNamedReplicationSlot(name, true);
> +
> + if (!slot)
> + {
> + GUC_check_errdetail("replication slot \"%s\" does not exist", name);
> + ok = false; break; }
> +
> + if (!SlotIsPhysical(slot))
> + {
> + GUC_check_errdetail("\"%s\" is not a physical replication slot",
> + name); ok = false; break; } }
>
> Won't the second check need protection via ReplicationSlotControlLock?
Yes, added.
>
> 2. In WaitForStandbyConfirmation(), we are anyway calling
> StandbyConfirmedFlush() before the actual sleep in the loop, so does calling it
> at the beginning of the function will serve any purpose? If so, it is better to add
> some comments explaining the same.
It is used as a fast-path to avoid calling condition variable stuff, I think we can directly
put failover check and list check in the beginning instead of calling that function.
>
> 3. Also do we need to perform the validation checks done in
> StandbyConfirmedFlush() repeatedly when it is invoked in a loop? We can
> probably separate those checks and perform them just once.
I have removed slot.failover check from the StandbyConfirmedFlush
function, so that we can do it when necessary. I didn’t change the check
for standby_slot_names_list because the list could be changed in the loop
when reloading config.
>
> 4.
> + *
> + * XXX: If needed, this can be attempted in future.
>
> Remove this part of the comment.
Removed.
Attach the V102 patch set which addressed Amit and Shveta's comments.
Thanks Shveta for helping addressing the comments off-list.
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v102-0002-Document-the-steps-to-check-if-the-standby-is-r.patch (7.0K, ../../OS0PR01MB5716E281A55E5053DF4976D1945E2@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v102-0002-Document-the-steps-to-check-if-the-standby-is-r.patch)
download | inline diff:
From d07ca9bc9528a90a7b5a6ce155b350abe177ae21 Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Fri, 19 Jan 2024 11:04:16 +0530
Subject: [PATCH v102 2/2] Document the steps to check if the standby is ready
for failover
---
doc/src/sgml/high-availability.sgml | 9 ++
doc/src/sgml/logical-replication.sgml | 136 ++++++++++++++++++++++++++
2 files changed, 145 insertions(+)
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 236c0af65f..36215aa68c 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -1487,6 +1487,15 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)'
Written administration procedures are advised.
</para>
+ <para>
+ If you have opted for synchronization of logical slots (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>),
+ then before switching to the standby server, it is recommended to check
+ if the logical slots synchronized on the standby server are ready
+ for failover. This can be done by following the steps described in
+ <xref linkend="logical-replication-failover"/>.
+ </para>
+
<para>
To trigger failover of a log-shipping standby server, run
<command>pg_ctl promote</command> or call <function>pg_promote()</function>.
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ec2130669e..be59d306a1 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -687,6 +687,142 @@ ALTER SUBSCRIPTION
</sect1>
+ <sect1 id="logical-replication-failover">
+ <title>Logical Replication Failover</title>
+
+ <para>
+ When the publisher server is the primary server of a streaming replication,
+ the logical slots on that primary server can be synchronized to the standby
+ server by specifying <literal>failover = true</literal> when creating
+ subscriptions for those publications. Enabling failover ensures a seamless
+ transition of those subscriptions after the standby is promoted. They can
+ continue subscribing to publications now on the new primary server without
+ any data loss.
+ </para>
+
+ <para>
+ Because the slot synchronization logic copies asynchronously, it is
+ necessary to confirm that replication slots have been synced to the standby
+ server before the failover happens. Furthermore, to ensure a successful
+ failover, the standby server must not be lagging behind the subscriber. It
+ is highly recommended to use
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ to prevent the subscriber from consuming changes faster than the hot standby.
+ To confirm that the standby server is indeed ready for failover, follow
+ these 2 steps:
+ </para>
+
+ <procedure>
+ <step performance="required">
+ <para>
+ Confirm that all the necessary logical replication slots have been synced to
+ the standby server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node, use the following SQL to identify
+ which slots should be synced to the standby that we plan to promote.
+<programlisting>
+test_sub=# SELECT
+ array_agg(slotname) AS slots
+ FROM
+ ((
+ SELECT r.srsubid AS subid, CONCAT('pg_', srsubid, '_sync_', srrelid, '_', ctl.system_identifier) AS slotname
+ FROM pg_control_system() ctl, pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate = 'f' AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT s.oid AS subid, s.subslotname as slotname
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ slots
+-------
+ {sub1,sub2,sub3}
+(1 row)
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, check that the logical replication slots identified above exist on
+ the standby server and are ready for failover.
+<programlisting>
+test_standby=# SELECT slot_name, (synced AND NOT temporary AND conflict_reason IS NULL) AS failover_ready
+ FROM pg_replication_slots
+ WHERE slot_name IN ('sub1','sub2','sub3');
+ slot_name | failover_ready
+-------------+----------------
+ sub1 | t
+ sub2 | t
+ sub3 | t
+(3 rows)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+
+ <step performance="required">
+ <para>
+ Confirm that the standby server is not lagging behind the subscribers.
+ This step can be skipped if
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ has been correctly configured. If standby_slot_names is not configured
+ correctly, it is highly recommended to run this step after the primary
+ server is down, otherwise the results of the query may vary at different
+ points of time due to the ongoing replication on the logical subscribers
+ from the primary server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node check the last replayed WAL.
+ This step needs to be run on the database(s) that includes the failover
+ enabled subscription(s), to find the last replayed WAL on each database.
+<programlisting>
+test_sub=# SELECT
+ MAX(remote_lsn) AS remote_lsn_on_subscriber
+ FROM
+ ((
+ SELECT (CASE WHEN r.srsubstate = 'f' THEN pg_replication_origin_progress(CONCAT('pg_', r.srsubid, '_', r.srrelid), false)
+ WHEN r.srsubstate IN ('s', 'r') THEN r.srsublsn END) AS remote_lsn
+ FROM pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate IN ('f', 's', 'r') AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT pg_replication_origin_progress(CONCAT('pg_', s.oid), false) AS remote_lsn
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ remote_lsn_on_subscriber
+--------------------------
+ 0/3000388
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, on the standby server check that the last-received WAL location
+ is ahead of the replayed WAL location(s) on the subscriber identified
+ above. If the above SQL result was NULL, it means the subscriber has not
+ yet replayed any WAL, so the standby server must be ahead of the
+ subscriber, and this step can be skipped.
+<programlisting>
+test_standby=# SELECT pg_last_wal_receive_lsn() >= '0/3000388'::pg_lsn AS failover_ready;
+ failover_ready
+----------------
+ t
+(1 row)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+ </procedure>
+
+ <para>
+ If the result (<literal>failover_ready</literal>) of both above steps is
+ true, existing subscriptions will be able to continue without data loss.
+ </para>
+
+ </sect1>
+
<sect1 id="logical-replication-row-filter">
<title>Row Filters</title>
--
2.30.0.windows.2
[application/octet-stream] v102-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch (41.9K, ../../OS0PR01MB5716E281A55E5053DF4976D1945E2@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v102-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch)
download | inline diff:
From 2dff694b61a4bf23010dc8f88d02731af0c4d95b Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Fri, 23 Feb 2024 08:29:30 +0800
Subject: [PATCH v102] Allow logical walsenders to wait for the physical
standby
This patch introduces a mechanism to ensure that physical standby servers,
which are potential failover candidates, have received and flushed changes
before making them visible to subscribers. By doing so, it guarantees that
the promoted standby server is not lagging behind the subscribers when a
failover is necessary.
A new parameter named 'standby_slot_names' is introduced. The logical
walsender now guarantees that all local changes are sent and flushed to
the standby servers corresponding to the replication slots specified in
'standby_slot_names' before sending those changes to the subscriber.
Additionally, The SQL functions pg_logical_slot_get_changes and
pg_replication_slot_advance are modified to wait for the replication slots
mentioned in 'standby_slot_names' to catch up before returning.
---
doc/src/sgml/config.sgml | 29 ++
doc/src/sgml/logicaldecoding.sgml | 12 +
.../replication/logical/logicalfuncs.c | 12 +
src/backend/replication/logical/slotsync.c | 11 +
src/backend/replication/slot.c | 370 +++++++++++++++++-
src/backend/replication/slotfuncs.c | 12 +
src/backend/replication/walsender.c | 135 ++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 14 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/replication/slot.h | 5 +
src/include/replication/walsender.h | 1 +
src/include/replication/walsender_private.h | 7 +
src/include/utils/guc_hooks.h | 3 +
src/test/recovery/t/006_logical_decoding.pl | 3 +-
.../t/040_standby_failover_slots_sync.pl | 202 ++++++++++
16 files changed, 799 insertions(+), 20 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 43b1a132a2..ea95561420 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4559,6 +4559,35 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Lists the streaming replication standby server slot names that logical
+ WAL sender processes will wait for. Logical WAL sender processes will
+ send decoded changes to plugins only after the specified replication
+ slots confirm receiving WAL. This guarantees that logical replication
+ slots with failover enabled do not consume changes until those changes
+ are received and flushed to corresponding physical standbys. If a
+ logical replication connection is meant to switch to a physical standby
+ after the standby is promoted, the physical replication slot for the
+ standby should be listed here. Note that logical replication will not
+ proceed if the slots specified in the standby_slot_names do not exist or
+ are invalidated.
+ </para>
+ <para>
+ The standbys corresponding to the physical replication slots in
+ <varname>standby_slot_names</varname> must configure
+ <literal>sync_replication_slots = true</literal> so they can receive
+ failover logical slots changes from the primary.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 930c0fa8a6..5da3ac7f4c 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -384,6 +384,18 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU
must be enabled on the standby. It is also necessary to specify a valid
<literal>dbname</literal> in the
<link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>.
+ It's highly recommended that the said physical replication slot is named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ list on the primary, to prevent the subscriber from consuming changes
+ faster than the hot standby. Even when correctly configured, some latency
+ is expected when sending changes to logical subscribers due to the waiting
+ on slots named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>.
+ When <varname>standby_slot_names</varname> is utilized, the
+ primary server will not completely shut down until the corresponding
+ standbys, associated with the physical replication slots specified
+ in <varname>standby_slot_names</varname>, have confirmed
+ receiving the WAL up to the latest flushed position on the primary server.
</para>
<para>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b0081d3ce5..f28d6c9015 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -109,6 +109,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
MemoryContext per_query_ctx;
MemoryContext oldcontext;
XLogRecPtr end_of_wal;
+ XLogRecPtr wait_for_wal_lsn;
LogicalDecodingContext *ctx;
ResourceOwner old_resowner = CurrentResourceOwner;
ArrayType *arr;
@@ -228,6 +229,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
NameStr(MyReplicationSlot->data.plugin),
format_procedure(fcinfo->flinfo->fn_oid))));
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to wait_for_wal_lsn.
+ */
+ if (XLogRecPtrIsInvalid(upto_lsn))
+ wait_for_wal_lsn = end_of_wal;
+ else
+ wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
+
+ WaitForStandbyConfirmation(wait_for_wal_lsn);
+
ctx->output_writer_private = p;
/*
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 36773cfe73..a1796c4ea6 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -491,6 +491,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
latestFlushPtr = GetStandbyFlushRecPtr(NULL);
if (remote_slot->confirmed_lsn > latestFlushPtr)
{
+ /*
+ * Can get here only if GUC 'standby_slot_names' on the primary server
+ * was not configured correctly.
+ */
ereport(am_slotsync_worker ? LOG : ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("skipping slot synchronization as the received slot sync"
@@ -860,6 +864,13 @@ validate_remote_info(WalReceiverConn *wrconn)
remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull));
Assert(!isnull);
+ /*
+ * Slot sync is currently not supported on a cascading standby. This is
+ * because if we allow it, the primary server needs to wait for all the
+ * cascading standbys, otherwise, logical subscribers can still be ahead
+ * of one of the cascading standbys which we plan to promote. Thus, to
+ * avoid this additional complexity, we restrict it for the time being.
+ */
if (remote_in_recovery)
ereport(ERROR,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 0f173f63a2..2f4bba2592 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -46,13 +46,18 @@
#include "common/string.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/interrupt.h"
#include "replication/slotsync.h"
#include "replication/slot.h"
+#include "replication/walsender_private.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
+#include "utils/memutils.h"
+#include "utils/varlena.h"
/*
* Replication slot on-disk data structure.
@@ -115,10 +120,25 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
/* My backend's replication slot in the shared memory array */
ReplicationSlot *MyReplicationSlot = NULL;
-/* GUC variable */
+/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+/*
+ * This GUC lists streaming replication standby server slot names that
+ * logical WAL sender processes will wait for.
+ */
+char *standby_slot_names;
+
+/* This is parsed and cached list for raw standby_slot_names. */
+static List *standby_slot_names_list = NIL;
+
+/*
+ * Oldest LSN that has been confirmed to be flushed to the standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+static XLogRecPtr standby_slot_oldest_flush_lsn = InvalidXLogRecPtr;
+
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
@@ -2345,3 +2365,351 @@ GetSlotInvalidationCause(const char *conflict_reason)
Assert(found);
return result;
}
+
+/*
+ * A helper function to validate slots specified in GUC standby_slot_names.
+ */
+static bool
+validate_standby_slots(char **newval)
+{
+ char *rawname;
+ List *elemlist;
+ bool ok;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Verify syntax and parse string into a list of identifiers */
+ ok = SplitIdentifierString(rawname, ',', &elemlist);
+
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
+ else if (!ReplicationSlotCtl)
+ {
+ /*
+ * We cannot validate the replication slot if the replication slots'
+ * data has not been initialized. This is ok as we will validate the
+ * specified slot when waiting for them to catch up. See
+ * StandbySlotsHaveCaughtup for details.
+ */
+ }
+ else
+ {
+ /*
+ * If the replication slots' data have been initialized, verify if the
+ * specified slots exist and are logical slots.
+ */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, elemlist)
+ {
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ GUC_check_errdetail("replication slot \"%s\" does not exist",
+ name);
+ ok = false;
+ break;
+ }
+
+ if (!SlotIsPhysical(slot))
+ {
+ GUC_check_errdetail("\"%s\" is not a physical replication slot",
+ name);
+ ok = false;
+ break;
+ }
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+ }
+
+ pfree(rawname);
+ list_free(elemlist);
+ return ok;
+}
+
+/*
+ * GUC check_hook for standby_slot_names
+ */
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ if ((*newval)[0] == '\0')
+ return true;
+
+ /*
+ * "*" is not accepted as in that case primary will not be able to know
+ * for which all standbys to wait for. Even if we have physical slots
+ * info, there is no way to confirm whether there is any standby
+ * configured for the known physical slots.
+ */
+ if (strcmp(*newval, "*") == 0)
+ {
+ GUC_check_errdetail("\"*\" is not accepted for standby_slot_names");
+ return false;
+ }
+
+ /* Now verify if the specified slots really exist and have correct type */
+ if (!validate_standby_slots(newval))
+ return false;
+
+ *extra = guc_strdup(ERROR, *newval);
+
+ return true;
+}
+
+/*
+ * GUC assign_hook for standby_slot_names
+ */
+void
+assign_standby_slot_names(const char *newval, void *extra)
+{
+ List *standby_slots;
+ MemoryContext oldcxt;
+ char *standby_slot_names_cpy = extra;
+
+ /*
+ * The standby slots may have changed, so we need to recompute the oldest
+ * LSN.
+ */
+ standby_slot_oldest_flush_lsn = InvalidXLogRecPtr;
+
+ list_free(standby_slot_names_list);
+ standby_slot_names_list = NIL;
+
+ /* No value is specified for standby_slot_names. */
+ if (standby_slot_names_cpy == NULL)
+ return;
+
+ if (!SplitIdentifierString(standby_slot_names_cpy, ',', &standby_slots))
+ {
+ /* This should not happen if GUC checked check_standby_slot_names. */
+ elog(ERROR, "invalid list syntax");
+ }
+
+ /*
+ * Switch to the memory context under which GUC variables are allocated
+ * (GUCMemoryContext).
+ */
+ oldcxt = MemoryContextSwitchTo(GetMemoryChunkContext(standby_slot_names_cpy));
+ standby_slot_names_list = list_copy(standby_slots);
+ MemoryContextSwitchTo(oldcxt);
+}
+
+/*
+ * Return the standby_slot_names_list.
+ *
+ * Note that since we do not support syncing slots to cascading standbys, we
+ * return NIL if we are running in a standby to indicate that no standby slots
+ * need to be waited for.
+ */
+List *
+GetStandbySlotList(void)
+{
+ if (RecoveryInProgress())
+ return NIL;
+ else
+ return standby_slot_names_list;
+}
+
+/*
+ * Return true if the slots specified in standby_slot_names have caught up to
+ * the given WAL location, false otherwise.
+ *
+ * The elevel parameter determines the error level used for logging messages
+ * related to slots that do not exist, are invalidated, or are inactive.
+ */
+bool
+StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel)
+{
+ int caught_up_slot_num = 0;
+ XLogRecPtr min_restart_lsn = InvalidXLogRecPtr;
+
+ /*
+ * Don't need to wait for the standby to catch up if there is no value in
+ * standby_slot_names.
+ */
+ if (!standby_slot_names_list)
+ return true;
+
+ /*
+ * If we are on a standby server, we do not need to wait for the standby to
+ * catch up since we do not support syncing slots to cascading standbys.
+ */
+ if (RecoveryInProgress())
+ return true;
+
+ /*
+ * Return true if all the standbys have already caught up to the passed in
+ * WAL localtion.
+ */
+ if (!XLogRecPtrIsInvalid(standby_slot_oldest_flush_lsn) &&
+ standby_slot_oldest_flush_lsn >= wait_for_lsn)
+ return true;
+
+ /*
+ * To prevent concurrent slot dropping and creation while filtering the
+ * slots, take the ReplicationSlotControlLock outside of the loop.
+ */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, standby_slot_names_list)
+ {
+ XLogRecPtr restart_lsn;
+ bool invalidated;
+ bool inactive;
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ /*
+ * It may happen that the slot specified in standby_slot_names GUC
+ * value is dropped, so let's skip over it.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not exist",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider creating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ continue;
+ }
+
+ if (SlotIsLogical(slot))
+ {
+ /*
+ * If a logical slot name is provided in standby_slot_names, report
+ * a message and skip it. It is possible for user to specify a
+ * logical slot name in standby_slot_names just before the server
+ * startup. The GUC check_hook(validate_standby_slots) can not
+ * validate such a slot during startup as the ReplicationSlotCtl
+ * shared memory is not initialized by that time. It is also
+ * possible for user to drop an existing physical slot and
+ * recreate a logical slot with the same name.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot have logical replication slot \"%s\" in parameter %s",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting for correction on \"%s\".",
+ name),
+ errhint("Consider removing logical slot \"%s\" from parameter %s.",
+ name, "standby_slot_names"));
+ continue;
+ }
+
+ SpinLockAcquire(&slot->mutex);
+ restart_lsn = slot->data.restart_lsn;
+ invalidated = slot->data.invalidated != RS_INVAL_NONE;
+ inactive = slot->active_pid == 0;
+ SpinLockRelease(&slot->mutex);
+
+ if (invalidated)
+ {
+ /* Specified physical slot has been invalidated */
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("physical slot \"%s\" specified in parameter %s has been invalidated",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider dropping and recreating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ continue;
+ }
+
+ if (XLogRecPtrIsInvalid(restart_lsn) || restart_lsn < wait_for_lsn)
+ {
+ /* Log a message if no active_pid for this physical slot */
+ if (inactive)
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not have active_pid",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider starting standby associated with \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+
+ /* Continue if the current slot hasn't caught up. */
+ continue;
+ }
+
+ Assert(restart_lsn >= wait_for_lsn);
+
+ if (XLogRecPtrIsInvalid(min_restart_lsn) ||
+ min_restart_lsn > restart_lsn)
+ min_restart_lsn = restart_lsn;
+
+ caught_up_slot_num++;
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /*
+ * Return false if not all the standbys have caught up to the specified WAL
+ * location.
+ */
+ if (caught_up_slot_num != list_length(standby_slot_names_list))
+ return false;
+
+ standby_slot_oldest_flush_lsn = min_restart_lsn;
+
+ return true;
+}
+
+/*
+ * Wait for physical standbys to confirm receiving the given lsn.
+ *
+ * Used by logical decoding SQL functions that acquired failover enabled slot.
+ * It waits for physical standbys corresponding to the physical slots specified
+ * in the standby_slot_names GUC.
+ */
+void
+WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
+{
+ /*
+ * Don't need to wait for the standby to catch up if the current acquired
+ * slot is not a failover enabled slot, or there is no value in
+ * standby_slot_names.
+ */
+ if (!MyReplicationSlot->data.failover || !standby_slot_names_list)
+ return;
+
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+
+ for (;;)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /* Exit if done waiting for every slot. */
+ if (StandbySlotsHaveCaughtup(wait_for_lsn, WARNING))
+ break;
+
+ /*
+ * We wait for the slots in the standby_slot_names to catch up, but we
+ * use a timeout (1s) so we can also check if the standby_slot_names
+ * has been changed.
+ */
+ ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000,
+ WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION);
+ }
+
+ ConditionVariableCancelSleep();
+}
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 768a304723..d864fe4133 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -464,6 +464,12 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* crash, but this makes the data consistent after a clean shutdown.
*/
ReplicationSlotMarkDirty();
+
+ /*
+ * Wake up logical walsenders holding failover enabled slots after
+ * updating the restart_lsn of the physical slot.
+ */
+ PhysicalWakeupLogicalWalSnd();
}
return retlsn;
@@ -504,6 +510,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto)
.segment_close = wal_segment_close),
NULL, NULL, NULL);
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to moveto lsn.
+ */
+ WaitForStandbyConfirmation(moveto);
+
/*
* Start reading at the slot's restart_lsn, which we know to point to
* a valid record.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 13bc3e0aee..0fbab3cb0c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1728,26 +1728,106 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
ProcessPendingWrites();
}
+/*
+ * Wake up the logical walsender processes with failover enabled slots if the
+ * currently acquired physical slot is specified in standby_slot_names GUC.
+ */
+void
+PhysicalWakeupLogicalWalSnd(void)
+{
+ List *standby_slots;
+
+ Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot));
+
+ standby_slots = GetStandbySlotList();
+
+ foreach_ptr(char, name, standby_slots)
+ {
+ if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0)
+ {
+ ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv);
+ return;
+ }
+ }
+}
+
+/*
+ * Returns true if there are not enough WALs to be read, or if not all standbys
+ * have caught up to the flushed position when failover_slot is true;
+ * otherwise, returns false.
+ *
+ * Set prioritize_stop to true to skip waiting for WALs if the shutdown signal
+ * is received.
+ *
+ * Set failover_slot to true if the current acquired slot is a failover enabled
+ * slot and we are streaming.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ bool prioritize_stop, bool failover_slot,
+ uint32 *wait_event)
+{
+ int elevel = got_STOPPING ? ERROR : WARNING;
+
+ *wait_event = 0;
+
+ /*
+ * Check if we need to wait for WALs to be flushed to disk. We don't need
+ * to wait for WALs after receiving the shutdown signal unless the
+ * wait_for_wal_on_stop is true.
+ */
+ if (target_lsn > flushed_lsn && !(prioritize_stop && got_STOPPING))
+ *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
+
+ /*
+ * Check if the standby slots have caught up to the flushed position. It is
+ * good to wait up to RecentFlushPtr and then let it send the changes to
+ * logical subscribers one by one which are already covered in
+ * RecentFlushPtr without needing to wait on every change for standby
+ * confirmation. Note that after receiving the shutdown signal, an ERROR is
+ * reported if any slots are dropped, invalidated, or inactive. This
+ * measure is taken to prevent the walsender from waiting indefinitely.
+ */
+ else if (failover_slot && !StandbySlotsHaveCaughtup(flushed_lsn, elevel))
+ *wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
+ else
+ return false;
+
+ return true;
+}
+
/*
* Wait till WAL < loc is flushed to disk so it can be safely sent to client.
*
- * Returns end LSN of flushed WAL. Normally this will be >= loc, but
- * if we detect a shutdown request (either from postmaster or client)
- * we will return early, so caller must always check.
+ * If the walsender holds a logical slot that has enabled failover, we also
+ * wait for all the specified streaming replication standby servers to
+ * confirm receipt of WAL up to RecentFlushPtr.
+ *
+ * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we
+ * detect a shutdown request (either from postmaster or client) we will return
+ * early, so caller must always check.
*/
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
int wakeEvents;
+ bool failover_slot;
+ uint32 wait_event;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
+ failover_slot = (replication_active && MyReplicationSlot->data.failover);
+
/*
* Fast path to avoid acquiring the spinlock in case we already know we
- * have enough WAL available. This is particularly interesting if we're
- * far behind.
+ * have enough WAL available and all the standby servers have confirmed
+ * receipt of WAL up to RecentFlushPtr. This is particularly interesting if
+ * we're far behind.
*/
- if (RecentFlushPtr != InvalidXLogRecPtr &&
- loc <= RecentFlushPtr)
+ if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, false, failover_slot, &wait_event))
return RecentFlushPtr;
/* Get a more recent flush pointer. */
@@ -1784,20 +1864,28 @@ WalSndWaitForWal(XLogRecPtr loc)
if (got_STOPPING)
XLogBackgroundFlush();
- /* Update our idea of the currently flushed position. */
- if (!RecoveryInProgress())
- RecentFlushPtr = GetFlushRecPtr(NULL);
- else
- RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * Update our idea of the currently flushed position only if we are
+ * waiting for more WALs.
+ */
+ if (wait_event == WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL)
+ {
+ if (!RecoveryInProgress())
+ RecentFlushPtr = GetFlushRecPtr(NULL);
+ else
+ RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ }
/*
- * If postmaster asked us to stop, don't wait anymore.
+ * If postmaster asked us to stop and the standby slots have caught up
+ * to the flushed position, don't wait anymore.
*
* It's important to do this check after the recomputation of
* RecentFlushPtr, so we can send all remaining data before shutting
* down.
*/
- if (got_STOPPING)
+ if (got_STOPPING && (!failover_slot ||
+ StandbySlotsHaveCaughtup(RecentFlushPtr, ERROR)))
break;
/*
@@ -1813,8 +1901,11 @@ WalSndWaitForWal(XLogRecPtr loc)
!waiting_for_ping_response)
WalSndKeepalive(false, InvalidXLogRecPtr);
- /* check whether we're done */
- if (loc <= RecentFlushPtr)
+ /*
+ * Exit the loop if already caught up and doesn't need to wait for
+ * standby slots.
+ */
+ if (!NeedToWaitForWal(loc, RecentFlushPtr, true, failover_slot, &wait_event))
break;
/* Waiting for new WAL. Since we need to wait, we're now caught up. */
@@ -1855,7 +1946,7 @@ WalSndWaitForWal(XLogRecPtr loc)
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL);
+ WalSndWait(wakeEvents, sleeptime, wait_event);
}
/* reactivate latch so WalSndLoop knows to continue */
@@ -2265,6 +2356,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn)
{
ReplicationSlotMarkDirty();
ReplicationSlotsComputeRequiredLSN();
+ PhysicalWakeupLogicalWalSnd();
}
/*
@@ -3538,6 +3630,7 @@ WalSndShmemInit(void)
ConditionVariableInit(&WalSndCtl->wal_flush_cv);
ConditionVariableInit(&WalSndCtl->wal_replay_cv);
+ ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv);
}
}
@@ -3607,8 +3700,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event)
*
* And, we use separate shared memory CVs for physical and logical
* walsenders for selective wake ups, see WalSndWakeup() for more details.
+ *
+ * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV
+ * until awakened by physical walsenders after the walreceiver confirms
+ * the receipt of the LSN.
*/
- if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
+ if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+ else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv);
else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index ec2f31f82a..c08e00d1d6 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP
LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server."
LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server."
SSL_OPEN_SERVER "Waiting for SSL while attempting connection."
+WAIT_FOR_STANDBY_CONFIRMATION "Waiting for WAL to be received and flushed by the physical standby."
WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process."
WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 93ded31ed9..326cefcf4b 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4670,6 +4670,20 @@ struct config_string ConfigureNamesString[] =
check_debug_io_direct, assign_debug_io_direct, NULL
},
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("Lists streaming replication standby server slot "
+ "names that logical WAL sender processes will wait for."),
+ gettext_noop("Logical WAL sender processes will send decoded "
+ "changes to plugins only after the specified "
+ "replication slots confirm receiving WAL."),
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, assign_standby_slot_names, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index edcc0282b2..2244ee52f7 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -343,6 +343,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsender processes will wait for
# - Standby Servers -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index acbf567150..61a6b97952 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -226,6 +226,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
+extern PGDLLIMPORT char *standby_slot_names;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -274,4 +275,8 @@ extern void CheckSlotPermissions(void);
extern ReplicationSlotInvalidationCause
GetSlotInvalidationCause(const char *conflict_reason);
+extern List *GetStandbySlotList(void);
+extern bool StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel);
+extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn);
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index 0c3996e926..f2d8297f01 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -39,6 +39,7 @@ extern void InitWalSender(void);
extern bool exec_replication_command(const char *cmd_string);
extern void WalSndErrorCleanup(void);
extern void WalSndResourceCleanup(bool isCommit);
+extern void PhysicalWakeupLogicalWalSnd(void);
extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
extern void WalSndSignals(void);
extern Size WalSndShmemSize(void);
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 3113e9ea47..3c134c8edf 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -113,6 +113,13 @@ typedef struct
ConditionVariable wal_flush_cv;
ConditionVariable wal_replay_cv;
+ /*
+ * Used by physical walsenders holding slots specified in
+ * standby_slot_names to wake up logical walsenders holding failover
+ * enabled slots when a walreceiver confirms the receipt of LSN.
+ */
+ ConditionVariable wal_confirm_rcv_cv;
+
WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER];
} WalSndCtlData;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index c8a7aa9a11..d64dc5fcdb 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra,
extern void assign_wal_consistency_checking(const char *newval, void *extra);
extern bool check_wal_segment_size(int *newval, void **extra, GucSource source);
extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
+extern bool check_standby_slot_names(char **newval, void **extra,
+ GucSource source);
+extern void assign_standby_slot_names(const char *newval, void *extra);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl
index 5c7b4ca5e3..b95d95c06f 100644
--- a/src/test/recovery/t/006_logical_decoding.pl
+++ b/src/test/recovery/t/006_logical_decoding.pl
@@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'},
undef, 'logical slot was actually dropped with DB');
# Test logical slot advancing and its durability.
+# Passing failover=true (last arg) should not have any impact on advancing.
my $logical_slot = 'logical_slot';
$node_primary->safe_psql('postgres',
- "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);"
+ "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);"
);
$node_primary->psql(
'postgres', "
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 968aa7b05b..4d0f28b79e 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -446,11 +446,213 @@ ok( $standby1->poll_query_until(
"SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot' AND synced AND NOT temporary;"),
'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby');
+##################################################
+# Test primary disallowing specified logical replication slots getting ahead of
+# specified physical replication slots. It uses the following set up:
+#
+# (physical standbys)
+# | ----> standby1 (primary_slot_name = sb1_slot)
+# | ----> standby2 (primary_slot_name = sb2_slot)
+# primary ----- |
+# (logical replication)
+# | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
+# | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
+#
+# standby_slot_names = 'sb1_slot'
+#
+# Set up is configured in such a way that the logical slot of subscriber1 is
+# enabled for failover, thus it will wait for the physical slot of
+# standby1(sb1_slot) to catch up before sending decoded changes to subscriber1.
+##################################################
+
+$backup_name = 'backup3';
+
+$primary->psql('postgres',
+ q{SELECT pg_create_physical_replication_slot('sb2_slot');});
+
+$primary->backup($backup_name);
+
+# Create another standby
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup(
+ $primary, $backup_name,
+ has_streaming => 1,
+ has_restoring => 1);
+$standby2->append_conf(
+ 'postgresql.conf', qq(
+primary_slot_name = 'sb2_slot'
+));
+$standby2->start;
+$primary->wait_for_replay_catchup($standby2);
+
+# Configure primary to disallow any logical slots that enabled failover from
+# getting ahead of specified physical replication slot (sb1_slot).
+$primary->append_conf(
+ 'postgresql.conf', qq(
+standby_slot_names = 'sb1_slot'
+));
+$primary->reload;
+
+# Create another subscriber node without enabling failover, wait for sync to
+# complete
+my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$subscriber2->init;
+$subscriber2->start;
+$subscriber2->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE tab_int (a int PRIMARY KEY);
+ CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot);
+]);
+
+$subscriber2->wait_for_subscription_sync;
+
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+# Stop the standby associated with the specified physical replication slot
+# (sb1_slot) so that the logical replication slot (lsub1_slot) won't receive
+# changes until the standby comes up.
+$standby1->stop;
+
+# Create some data on the primary
+my $primary_row_count = 20;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(11, $primary_row_count);");
+
+# Wait until the standby2 that's still running gets the data from the primary
+$primary->wait_for_replay_catchup($standby2);
+$result = $standby2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby2 gets data from primary");
+
+# Wait for regress_mysub2 to get the data from the primary. This subscription
+# was not enabled for failover so it gets the data without waiting for any
+# standbys.
+$primary->wait_for_catchup('regress_mysub2');
+$result = $subscriber2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "subscriber2 gets data from primary");
+
+# The regress_mysub1 was enabled for failover so it doesn't get the data from
+# primary and keeps waiting for the standby specified in standby_slot_names
+# (sb1_slot aka standby1).
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) <> $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data from primary until standby1 acknowledges changes"
+);
+
+# Start the standby specified in standby_slot_names (sb1_slot aka standby1) and
+# wait for it to catch up with the primary.
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+$result = $standby1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby1 gets data from primary");
+
+# Now that the standby specified in standby_slot_names is up and running, the
+# primary can send the decoded changes to the subscription enabled for failover
+# (i.e. regress_mysub1). While the standby was down, regress_mysub1 didn't
+# receive any data from the primary. i.e. the primary didn't allow it to go
+# ahead of standby.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 acknowledges changes");
+
+##################################################
+# Verify that when using pg_logical_slot_get_changes to consume changes from a
+# logical slot with failover enabled, it will also wait for the slots specified
+# in standby_slot_names to catch up.
+##################################################
+
+# Stop the standby associated with the specified physical replication slot so
+# that the logical replication slot won't receive changes until the standby
+# slot's restart_lsn is advanced or the slot is removed from the
+# standby_slot_names list.
+$primary->safe_psql('postgres', "TRUNCATE tab_int;");
+$primary->wait_for_catchup('regress_mysub1');
+$standby1->stop;
+
+# Create a logical 'test_decoding' replication slot with failover enabled
+$primary->safe_psql('postgres',
+ "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);"
+);
+
+my $back_q = $primary->background_psql(
+ 'postgres',
+ on_error_stop => 0,
+ timeout => $PostgreSQL::Test::Utils::timeout_default);
+
+# Try and get changes from the logical slot with failover enabled.
+my $offset = -s $primary->logfile;
+$back_q->query_until(qr//,
+ "SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);\n");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the function
+# pg_logical_slot_get_changes should now return, and the session can be
+# stopped.
+$back_q->quit;
+
+$primary->safe_psql('postgres',
+ "SELECT pg_drop_replication_slot('test_slot');"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
+##################################################
+# Test that logical replication will wait for the user-created inactive
+# physical slot to catch up until we remove the slot from standby_slot_names.
+##################################################
+
+# Create some data on the primary
+$primary_row_count = 10;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);");
+
+# The regress_mysub1 doesn't get the data from primary because the specified
+# standby slot (sb1_slot) in standby_slot_names is inactive.
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data as the sb1_slot doesn't catch up");
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the primary server should now
+# send the decoded changes to the subscription.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list"
+);
+
##################################################
# Promote the standby1 to primary. Confirm that:
# a) the slot 'lsub1_slot' is retained on the new primary
# b) logical replication for regress_mysub1 is resumed successfully after failover
##################################################
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+
$standby1->promote;
# Update subscription with the new primary's connection info
--
2.30.0.windows.2
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-01 02:18 Zhijie Hou (Fujitsu) <[email protected]>
parent: Amit Kapila <[email protected]>
1 sibling, 0 replies; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-01 02:18 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; Peter Smith <[email protected]>; +Cc: shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Thursday, February 29, 2024 5:54 PM Amit Kapila <[email protected]> wrote:
>
> On Thu, Feb 29, 2024 at 8:29 AM Peter Smith <[email protected]>
> wrote:
> >
> > On Wed, Feb 28, 2024 at 1:23 PM Zhijie Hou (Fujitsu)
> > <[email protected]> wrote:
> > >
> > > On Tuesday, February 27, 2024 3:18 PM Peter Smith
> <[email protected]> wrote:
> > ...
> > > > 20.
> > > > +#
> > > > +# | ----> standby1 (primary_slot_name = sb1_slot) # | ----> standby2
> > > > +(primary_slot_name = sb2_slot) # primary ----- | # | ----> subscriber1
> > > > +(failover = true) # | ----> subscriber2 (failover = false)
> > > >
> > > > In the diagram, the "--->" means a mixture of physical standbys and
> logical
> > > > pub/sub replication. Maybe it can be a bit clearer?
> > > >
> > > > SUGGESTION
> > > >
> > > > # primary (publisher)
> > > > #
> > > > # (physical standbys)
> > > > # | ----> standby1 (primary_slot_name = sb1_slot)
> > > > # | ----> standby2 (primary_slot_name = sb2_slot)
> > > > #
> > > > # (logical replication)
> > > > # | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
> > > > # | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
> > > >
> > >
> > > I think one can distinguish it based on the 'standby' and 'subscriber' as well,
> because
> > > 'standby' normally refer to physical standby while the other refer to logical.
> > >
>
> I think Peter's suggestion will make the setup clear.
Changed.
>
> >
> > Ok, but shouldn't it at least include info about the logical slot
> > names associated with the subscribers (slot_name = lsub1_slot,
> > slot_name = lsub2_slot) like suggested above?
> >
> > ======
> >
> > Here are some more review comments for v100-0001
> >
> > ======
> > doc/src/sgml/config.sgml
> >
> > 1.
> > + <para>
> > + Lists the streaming replication standby server slot names that
> logical
> > + WAL sender processes will wait for. Logical WAL sender processes
> will
> > + send decoded changes to plugins only after the specified
> replication
> > + slots confirm receiving WAL. This guarantees that logical replication
> > + slots with failover enabled do not consume changes until those
> changes
> > + are received and flushed to corresponding physical standbys. If a
> > + logical replication connection is meant to switch to a physical
> standby
> > + after the standby is promoted, the physical replication slot for the
> > + standby should be listed here. Note that logical replication will not
> > + proceed if the slots specified in the standby_slot_names do
> > not exist or
> > + are invalidated.
> > + </para>
> >
> > Is that note ("Note that logical replication will not proceed if the
> > slots specified in the standby_slot_names do not exist or are
> > invalidated") meant only for subscriptions marked for 'failover' or
> > any subscription? Maybe wording can be modified to help clarify it?
> >
>
> I think it is implicit that here we are talking about failover slots.
> I think clarifying again the same could be repetitive considering the
> previous sentence: "This guarantees that logical replication slots
> with failover enabled do not consume .." have mentioned it.
>
> > ======
> > src/backend/replication/slot.c
> >
> > 2.
> > +/*
> > + * A helper function to validate slots specified in GUC standby_slot_names.
> > + */
> > +static bool
> > +validate_standby_slots(char **newval)
> > +{
> > + char *rawname;
> > + List *elemlist;
> > + bool ok;
> > +
> > + /* Need a modifiable copy of string */
> > + rawname = pstrdup(*newval);
> > +
> > + /* Verify syntax and parse string into a list of identifiers */
> > + ok = SplitIdentifierString(rawname, ',', &elemlist);
> > +
> > + if (!ok)
> > + {
> > + GUC_check_errdetail("List syntax is invalid.");
> > + }
> > +
> > + /*
> > + * If the replication slots' data have been initialized, verify if the
> > + * specified slots exist and are logical slots.
> > + */
> > + else if (ReplicationSlotCtl)
> > + {
> > + foreach_ptr(char, name, elemlist)
> > + {
> > + ReplicationSlot *slot;
> > +
> > + slot = SearchNamedReplicationSlot(name, true);
> > +
> > + if (!slot)
> > + {
> > + GUC_check_errdetail("replication slot \"%s\" does not exist",
> > + name);
> > + ok = false;
> > + break;
> > + }
> > +
> > + if (!SlotIsPhysical(slot))
> > + {
> > + GUC_check_errdetail("\"%s\" is not a physical replication slot",
> > + name);
> > + ok = false;
> > + break;
> > + }
> > + }
> > + }
> > +
> > + pfree(rawname);
> > + list_free(elemlist);
> > + return ok;
> > +}
> >
> > 2a.
> > I didn't mention this previously because I thought this function was
> > not going to change anymore, but since Bertrand suggested some changes
> > [1], I will say IMO the { } are fine here for the single statement,
> > but I think it will be better to rearrange this code to be like below.
> > Having a 2nd NOP 'else' gives a much better place where you can put
> > your ReplicationSlotCtl comment.
> >
> > if (!ok)
> > {
> > GUC_check_errdetail("List syntax is invalid.");
> > }
> > else if (!ReplicationSlotCtl)
> > {
> > <Insert big comment here about why it is OK to skip when
> > ReplicationSlotCtl is NULL>
> > }
> > else
> > {
> > foreach_ptr ...
> > }
> >
>
> +1. This will make the code and reasoning to skip clear.
Changed.
>
> Few additional comments on the latest patch:
> =================================
> 1.
> static XLogRecPtr
> WalSndWaitForWal(XLogRecPtr loc)
> {
> ...
> + if (!XLogRecPtrIsInvalid(RecentFlushPtr) && loc <= RecentFlushPtr &&
> + (!replication_active || StandbyConfirmedFlush(loc, WARNING)))
> + {
> + /*
> + * Fast path to avoid acquiring the spinlock in case we already know
> + * we have enough WAL available and all the standby servers have
> + * confirmed receipt of WAL up to RecentFlushPtr. This is particularly
> + * interesting if we're far behind.
> + */
> return RecentFlushPtr;
> + }
> ...
> ...
> + * Wait for WALs to be flushed to disk only if the postmaster has not
> + * asked us to stop.
> + */
> + if (loc > RecentFlushPtr && !got_STOPPING)
> + wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
> +
> + /*
> + * Check if the standby slots have caught up to the flushed position.
> + * It is good to wait up to RecentFlushPtr and then let it send the
> + * changes to logical subscribers one by one which are already covered
> + * in RecentFlushPtr without needing to wait on every change for
> + * standby confirmation. Note that after receiving the shutdown signal,
> + * an ERROR is reported if any slots are dropped, invalidated, or
> + * inactive. This measure is taken to prevent the walsender from
> + * waiting indefinitely.
> + */
> + else if (replication_active &&
> + !StandbyConfirmedFlush(RecentFlushPtr, WARNING))
> + {
> + wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
> + wait_for_standby = true;
> + }
> + else
> + {
> + /* Already caught up and doesn't need to wait for standby_slots. */
> break;
> + }
> ...
> }
>
> Can we try to move these checks into a separate function that returns
> a boolean and has an out parameter as wait_event?
Refactored.
>
> 2. How about naming StandbyConfirmedFlush() as StandbySlotsAreCaughtup?
I used a similar version based on the suggested name: StandbySlotsHaveCaughtup.
Best Regards,
Hou zj
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-01 03:42 Zhijie Hou (Fujitsu) <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
1 sibling, 1 reply; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-01 03:42 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Friday, March 1, 2024 10:17 AM Zhijie Hou (Fujitsu) <[email protected]> wrote:
>
>
> Attach the V102 patch set which addressed Amit and Shveta's comments.
> Thanks Shveta for helping addressing the comments off-list.
The cfbot reported a compile warning, here is the new version patch which fixed it,
Also removed some outdate comments in this version.
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v102_2-0001-Allow-logical-walsenders-to-wait-for-the-physi.patch (42.2K, ../../OS0PR01MB571669D7F110110F7D87C48D945E2@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v102_2-0001-Allow-logical-walsenders-to-wait-for-the-physi.patch)
download | inline diff:
From cc8e9fe807ea60c5ff6bb226ee2fbeb7b9c4d349 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Fri, 23 Feb 2024 08:29:30 +0800
Subject: [PATCH v1022] Allow logical walsenders to wait for the physical
standby
This patch introduces a mechanism to ensure that physical standby servers,
which are potential failover candidates, have received and flushed changes
before making them visible to subscribers. By doing so, it guarantees that
the promoted standby server is not lagging behind the subscribers when a
failover is necessary.
A new parameter named 'standby_slot_names' is introduced. The logical
walsender now guarantees that all local changes are sent and flushed to
the standby servers corresponding to the replication slots specified in
'standby_slot_names' before sending those changes to the subscriber.
Additionally, The SQL functions pg_logical_slot_get_changes and
pg_replication_slot_advance are modified to wait for the replication slots
mentioned in 'standby_slot_names' to catch up before returning.
---
doc/src/sgml/config.sgml | 29 ++
doc/src/sgml/logicaldecoding.sgml | 12 +
.../replication/logical/logicalfuncs.c | 12 +
src/backend/replication/logical/slotsync.c | 11 +
src/backend/replication/slot.c | 370 +++++++++++++++++-
src/backend/replication/slotfuncs.c | 12 +
src/backend/replication/walsender.c | 139 ++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 14 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/replication/slot.h | 5 +
src/include/replication/walsender.h | 1 +
src/include/replication/walsender_private.h | 7 +
src/include/utils/guc_hooks.h | 3 +
src/test/recovery/t/006_logical_decoding.pl | 3 +-
.../t/040_standby_failover_slots_sync.pl | 202 ++++++++++
16 files changed, 802 insertions(+), 21 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 43b1a132a2..ea95561420 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4559,6 +4559,35 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Lists the streaming replication standby server slot names that logical
+ WAL sender processes will wait for. Logical WAL sender processes will
+ send decoded changes to plugins only after the specified replication
+ slots confirm receiving WAL. This guarantees that logical replication
+ slots with failover enabled do not consume changes until those changes
+ are received and flushed to corresponding physical standbys. If a
+ logical replication connection is meant to switch to a physical standby
+ after the standby is promoted, the physical replication slot for the
+ standby should be listed here. Note that logical replication will not
+ proceed if the slots specified in the standby_slot_names do not exist or
+ are invalidated.
+ </para>
+ <para>
+ The standbys corresponding to the physical replication slots in
+ <varname>standby_slot_names</varname> must configure
+ <literal>sync_replication_slots = true</literal> so they can receive
+ failover logical slots changes from the primary.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 930c0fa8a6..5da3ac7f4c 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -384,6 +384,18 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU
must be enabled on the standby. It is also necessary to specify a valid
<literal>dbname</literal> in the
<link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>.
+ It's highly recommended that the said physical replication slot is named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ list on the primary, to prevent the subscriber from consuming changes
+ faster than the hot standby. Even when correctly configured, some latency
+ is expected when sending changes to logical subscribers due to the waiting
+ on slots named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>.
+ When <varname>standby_slot_names</varname> is utilized, the
+ primary server will not completely shut down until the corresponding
+ standbys, associated with the physical replication slots specified
+ in <varname>standby_slot_names</varname>, have confirmed
+ receiving the WAL up to the latest flushed position on the primary server.
</para>
<para>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b0081d3ce5..f28d6c9015 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -109,6 +109,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
MemoryContext per_query_ctx;
MemoryContext oldcontext;
XLogRecPtr end_of_wal;
+ XLogRecPtr wait_for_wal_lsn;
LogicalDecodingContext *ctx;
ResourceOwner old_resowner = CurrentResourceOwner;
ArrayType *arr;
@@ -228,6 +229,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
NameStr(MyReplicationSlot->data.plugin),
format_procedure(fcinfo->flinfo->fn_oid))));
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to wait_for_wal_lsn.
+ */
+ if (XLogRecPtrIsInvalid(upto_lsn))
+ wait_for_wal_lsn = end_of_wal;
+ else
+ wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
+
+ WaitForStandbyConfirmation(wait_for_wal_lsn);
+
ctx->output_writer_private = p;
/*
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 8ecb85b86a..d1562cf7b3 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -491,6 +491,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
latestFlushPtr = GetStandbyFlushRecPtr(NULL);
if (remote_slot->confirmed_lsn > latestFlushPtr)
{
+ /*
+ * Can get here only if GUC 'standby_slot_names' on the primary server
+ * was not configured correctly.
+ */
ereport(am_slotsync_worker ? LOG : ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("skipping slot synchronization as the received slot sync"
@@ -860,6 +864,13 @@ validate_remote_info(WalReceiverConn *wrconn)
remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull));
Assert(!isnull);
+ /*
+ * Slot sync is currently not supported on a cascading standby. This is
+ * because if we allow it, the primary server needs to wait for all the
+ * cascading standbys, otherwise, logical subscribers can still be ahead
+ * of one of the cascading standbys which we plan to promote. Thus, to
+ * avoid this additional complexity, we restrict it for the time being.
+ */
if (remote_in_recovery)
ereport(ERROR,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 0f173f63a2..2f4bba2592 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -46,13 +46,18 @@
#include "common/string.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/interrupt.h"
#include "replication/slotsync.h"
#include "replication/slot.h"
+#include "replication/walsender_private.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
+#include "utils/memutils.h"
+#include "utils/varlena.h"
/*
* Replication slot on-disk data structure.
@@ -115,10 +120,25 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
/* My backend's replication slot in the shared memory array */
ReplicationSlot *MyReplicationSlot = NULL;
-/* GUC variable */
+/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+/*
+ * This GUC lists streaming replication standby server slot names that
+ * logical WAL sender processes will wait for.
+ */
+char *standby_slot_names;
+
+/* This is parsed and cached list for raw standby_slot_names. */
+static List *standby_slot_names_list = NIL;
+
+/*
+ * Oldest LSN that has been confirmed to be flushed to the standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+static XLogRecPtr standby_slot_oldest_flush_lsn = InvalidXLogRecPtr;
+
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
@@ -2345,3 +2365,351 @@ GetSlotInvalidationCause(const char *conflict_reason)
Assert(found);
return result;
}
+
+/*
+ * A helper function to validate slots specified in GUC standby_slot_names.
+ */
+static bool
+validate_standby_slots(char **newval)
+{
+ char *rawname;
+ List *elemlist;
+ bool ok;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Verify syntax and parse string into a list of identifiers */
+ ok = SplitIdentifierString(rawname, ',', &elemlist);
+
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
+ else if (!ReplicationSlotCtl)
+ {
+ /*
+ * We cannot validate the replication slot if the replication slots'
+ * data has not been initialized. This is ok as we will validate the
+ * specified slot when waiting for them to catch up. See
+ * StandbySlotsHaveCaughtup for details.
+ */
+ }
+ else
+ {
+ /*
+ * If the replication slots' data have been initialized, verify if the
+ * specified slots exist and are logical slots.
+ */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, elemlist)
+ {
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ GUC_check_errdetail("replication slot \"%s\" does not exist",
+ name);
+ ok = false;
+ break;
+ }
+
+ if (!SlotIsPhysical(slot))
+ {
+ GUC_check_errdetail("\"%s\" is not a physical replication slot",
+ name);
+ ok = false;
+ break;
+ }
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+ }
+
+ pfree(rawname);
+ list_free(elemlist);
+ return ok;
+}
+
+/*
+ * GUC check_hook for standby_slot_names
+ */
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ if ((*newval)[0] == '\0')
+ return true;
+
+ /*
+ * "*" is not accepted as in that case primary will not be able to know
+ * for which all standbys to wait for. Even if we have physical slots
+ * info, there is no way to confirm whether there is any standby
+ * configured for the known physical slots.
+ */
+ if (strcmp(*newval, "*") == 0)
+ {
+ GUC_check_errdetail("\"*\" is not accepted for standby_slot_names");
+ return false;
+ }
+
+ /* Now verify if the specified slots really exist and have correct type */
+ if (!validate_standby_slots(newval))
+ return false;
+
+ *extra = guc_strdup(ERROR, *newval);
+
+ return true;
+}
+
+/*
+ * GUC assign_hook for standby_slot_names
+ */
+void
+assign_standby_slot_names(const char *newval, void *extra)
+{
+ List *standby_slots;
+ MemoryContext oldcxt;
+ char *standby_slot_names_cpy = extra;
+
+ /*
+ * The standby slots may have changed, so we need to recompute the oldest
+ * LSN.
+ */
+ standby_slot_oldest_flush_lsn = InvalidXLogRecPtr;
+
+ list_free(standby_slot_names_list);
+ standby_slot_names_list = NIL;
+
+ /* No value is specified for standby_slot_names. */
+ if (standby_slot_names_cpy == NULL)
+ return;
+
+ if (!SplitIdentifierString(standby_slot_names_cpy, ',', &standby_slots))
+ {
+ /* This should not happen if GUC checked check_standby_slot_names. */
+ elog(ERROR, "invalid list syntax");
+ }
+
+ /*
+ * Switch to the memory context under which GUC variables are allocated
+ * (GUCMemoryContext).
+ */
+ oldcxt = MemoryContextSwitchTo(GetMemoryChunkContext(standby_slot_names_cpy));
+ standby_slot_names_list = list_copy(standby_slots);
+ MemoryContextSwitchTo(oldcxt);
+}
+
+/*
+ * Return the standby_slot_names_list.
+ *
+ * Note that since we do not support syncing slots to cascading standbys, we
+ * return NIL if we are running in a standby to indicate that no standby slots
+ * need to be waited for.
+ */
+List *
+GetStandbySlotList(void)
+{
+ if (RecoveryInProgress())
+ return NIL;
+ else
+ return standby_slot_names_list;
+}
+
+/*
+ * Return true if the slots specified in standby_slot_names have caught up to
+ * the given WAL location, false otherwise.
+ *
+ * The elevel parameter determines the error level used for logging messages
+ * related to slots that do not exist, are invalidated, or are inactive.
+ */
+bool
+StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel)
+{
+ int caught_up_slot_num = 0;
+ XLogRecPtr min_restart_lsn = InvalidXLogRecPtr;
+
+ /*
+ * Don't need to wait for the standby to catch up if there is no value in
+ * standby_slot_names.
+ */
+ if (!standby_slot_names_list)
+ return true;
+
+ /*
+ * If we are on a standby server, we do not need to wait for the standby to
+ * catch up since we do not support syncing slots to cascading standbys.
+ */
+ if (RecoveryInProgress())
+ return true;
+
+ /*
+ * Return true if all the standbys have already caught up to the passed in
+ * WAL localtion.
+ */
+ if (!XLogRecPtrIsInvalid(standby_slot_oldest_flush_lsn) &&
+ standby_slot_oldest_flush_lsn >= wait_for_lsn)
+ return true;
+
+ /*
+ * To prevent concurrent slot dropping and creation while filtering the
+ * slots, take the ReplicationSlotControlLock outside of the loop.
+ */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, standby_slot_names_list)
+ {
+ XLogRecPtr restart_lsn;
+ bool invalidated;
+ bool inactive;
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ /*
+ * It may happen that the slot specified in standby_slot_names GUC
+ * value is dropped, so let's skip over it.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not exist",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider creating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ continue;
+ }
+
+ if (SlotIsLogical(slot))
+ {
+ /*
+ * If a logical slot name is provided in standby_slot_names, report
+ * a message and skip it. It is possible for user to specify a
+ * logical slot name in standby_slot_names just before the server
+ * startup. The GUC check_hook(validate_standby_slots) can not
+ * validate such a slot during startup as the ReplicationSlotCtl
+ * shared memory is not initialized by that time. It is also
+ * possible for user to drop an existing physical slot and
+ * recreate a logical slot with the same name.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot have logical replication slot \"%s\" in parameter %s",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting for correction on \"%s\".",
+ name),
+ errhint("Consider removing logical slot \"%s\" from parameter %s.",
+ name, "standby_slot_names"));
+ continue;
+ }
+
+ SpinLockAcquire(&slot->mutex);
+ restart_lsn = slot->data.restart_lsn;
+ invalidated = slot->data.invalidated != RS_INVAL_NONE;
+ inactive = slot->active_pid == 0;
+ SpinLockRelease(&slot->mutex);
+
+ if (invalidated)
+ {
+ /* Specified physical slot has been invalidated */
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("physical slot \"%s\" specified in parameter %s has been invalidated",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider dropping and recreating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ continue;
+ }
+
+ if (XLogRecPtrIsInvalid(restart_lsn) || restart_lsn < wait_for_lsn)
+ {
+ /* Log a message if no active_pid for this physical slot */
+ if (inactive)
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not have active_pid",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider starting standby associated with \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+
+ /* Continue if the current slot hasn't caught up. */
+ continue;
+ }
+
+ Assert(restart_lsn >= wait_for_lsn);
+
+ if (XLogRecPtrIsInvalid(min_restart_lsn) ||
+ min_restart_lsn > restart_lsn)
+ min_restart_lsn = restart_lsn;
+
+ caught_up_slot_num++;
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /*
+ * Return false if not all the standbys have caught up to the specified WAL
+ * location.
+ */
+ if (caught_up_slot_num != list_length(standby_slot_names_list))
+ return false;
+
+ standby_slot_oldest_flush_lsn = min_restart_lsn;
+
+ return true;
+}
+
+/*
+ * Wait for physical standbys to confirm receiving the given lsn.
+ *
+ * Used by logical decoding SQL functions that acquired failover enabled slot.
+ * It waits for physical standbys corresponding to the physical slots specified
+ * in the standby_slot_names GUC.
+ */
+void
+WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
+{
+ /*
+ * Don't need to wait for the standby to catch up if the current acquired
+ * slot is not a failover enabled slot, or there is no value in
+ * standby_slot_names.
+ */
+ if (!MyReplicationSlot->data.failover || !standby_slot_names_list)
+ return;
+
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+
+ for (;;)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /* Exit if done waiting for every slot. */
+ if (StandbySlotsHaveCaughtup(wait_for_lsn, WARNING))
+ break;
+
+ /*
+ * We wait for the slots in the standby_slot_names to catch up, but we
+ * use a timeout (1s) so we can also check if the standby_slot_names
+ * has been changed.
+ */
+ ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000,
+ WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION);
+ }
+
+ ConditionVariableCancelSleep();
+}
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 768a304723..d864fe4133 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -464,6 +464,12 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* crash, but this makes the data consistent after a clean shutdown.
*/
ReplicationSlotMarkDirty();
+
+ /*
+ * Wake up logical walsenders holding failover enabled slots after
+ * updating the restart_lsn of the physical slot.
+ */
+ PhysicalWakeupLogicalWalSnd();
}
return retlsn;
@@ -504,6 +510,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto)
.segment_close = wal_segment_close),
NULL, NULL, NULL);
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to moveto lsn.
+ */
+ WaitForStandbyConfirmation(moveto);
+
/*
* Start reading at the slot's restart_lsn, which we know to point to
* a valid record.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 13bc3e0aee..5a64bf60b7 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1728,26 +1728,104 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
ProcessPendingWrites();
}
+/*
+ * Wake up the logical walsender processes with failover enabled slots if the
+ * currently acquired physical slot is specified in standby_slot_names GUC.
+ */
+void
+PhysicalWakeupLogicalWalSnd(void)
+{
+ List *standby_slots;
+
+ Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot));
+
+ standby_slots = GetStandbySlotList();
+
+ foreach_ptr(char, name, standby_slots)
+ {
+ if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0)
+ {
+ ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv);
+ return;
+ }
+ }
+}
+
+/*
+ * Returns true if the WALs before the target_lsn has been flushed to the disk,
+ * or if not all standbys have caught up to the flushed position (flushed_lsn)
+ * when failover_slot is true; otherwise, returns false.
+ *
+ * Set prioritize_stop to true to skip waiting for WALs if the shutdown signal
+ * is received.
+ *
+ * Set failover_slot to true if the current acquired slot is a failover enabled
+ * slot and we are streaming.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ bool prioritize_stop, bool failover_slot,
+ uint32 *wait_event)
+{
+ int elevel = got_STOPPING ? ERROR : WARNING;
+
+ /*
+ * Check if we need to wait for WALs to be flushed to disk. We don't need
+ * to wait for WALs after receiving the shutdown signal unless the
+ * prioritize_stop is false.
+ */
+ if (target_lsn > flushed_lsn && !(prioritize_stop && got_STOPPING))
+ *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
+
+ /*
+ * Check if the standby slots have caught up to the flushed position. It is
+ * good to wait up to flushed position and then let it send the changes to
+ * logical subscribers one by one which are already covered in flushed
+ * position without needing to wait on every change for standby
+ * confirmation. Note that after receiving the shutdown signal, an ERROR is
+ * reported if any slots are dropped, invalidated, or inactive. This
+ * measure is taken to prevent the walsender from waiting indefinitely.
+ */
+ else if (failover_slot && !StandbySlotsHaveCaughtup(flushed_lsn, elevel))
+ *wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
+ else
+ *wait_event = 0;
+
+ return *wait_event != 0;
+}
+
/*
* Wait till WAL < loc is flushed to disk so it can be safely sent to client.
*
- * Returns end LSN of flushed WAL. Normally this will be >= loc, but
- * if we detect a shutdown request (either from postmaster or client)
- * we will return early, so caller must always check.
+ * If the walsender holds a logical slot that has enabled failover, we also
+ * wait for all the specified streaming replication standby servers to
+ * confirm receipt of WAL up to RecentFlushPtr.
+ *
+ * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we
+ * detect a shutdown request (either from postmaster or client) we will return
+ * early, so caller must always check.
*/
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
int wakeEvents;
+ bool failover_slot;
+ uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
+ failover_slot = (replication_active && MyReplicationSlot->data.failover);
+
/*
* Fast path to avoid acquiring the spinlock in case we already know we
- * have enough WAL available. This is particularly interesting if we're
- * far behind.
+ * have enough WAL available and all the standby servers have confirmed
+ * receipt of WAL up to RecentFlushPtr. This is particularly interesting if
+ * we're far behind.
*/
- if (RecentFlushPtr != InvalidXLogRecPtr &&
- loc <= RecentFlushPtr)
+ if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, false, failover_slot, &wait_event))
return RecentFlushPtr;
/* Get a more recent flush pointer. */
@@ -1784,20 +1862,29 @@ WalSndWaitForWal(XLogRecPtr loc)
if (got_STOPPING)
XLogBackgroundFlush();
- /* Update our idea of the currently flushed position. */
- if (!RecoveryInProgress())
- RecentFlushPtr = GetFlushRecPtr(NULL);
- else
- RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * Update our idea of the currently flushed position only if we are not
+ * waiting for standbys to catch up, otherwise the standby would have
+ * to catch up to a newer WAL location in each cycle.
+ */
+ if (wait_event != WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ {
+ if (!RecoveryInProgress())
+ RecentFlushPtr = GetFlushRecPtr(NULL);
+ else
+ RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ }
/*
- * If postmaster asked us to stop, don't wait anymore.
+ * If postmaster asked us to stop and the standby slots have caught up
+ * to the flushed position, don't wait anymore.
*
* It's important to do this check after the recomputation of
* RecentFlushPtr, so we can send all remaining data before shutting
* down.
*/
- if (got_STOPPING)
+ if (got_STOPPING && (!failover_slot ||
+ StandbySlotsHaveCaughtup(RecentFlushPtr, ERROR)))
break;
/*
@@ -1813,11 +1900,17 @@ WalSndWaitForWal(XLogRecPtr loc)
!waiting_for_ping_response)
WalSndKeepalive(false, InvalidXLogRecPtr);
- /* check whether we're done */
- if (loc <= RecentFlushPtr)
+ /*
+ * Exit the loop if already caught up and doesn't need to wait for
+ * standby slots.
+ */
+ if (!NeedToWaitForWal(loc, RecentFlushPtr, true, failover_slot, &wait_event))
break;
- /* Waiting for new WAL. Since we need to wait, we're now caught up. */
+ /*
+ * Waiting for new WAL or waiting for standbys to catch up. Since
+ * we need to wait, we're now caught up.
+ */
WalSndCaughtUp = true;
/*
@@ -1855,7 +1948,7 @@ WalSndWaitForWal(XLogRecPtr loc)
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL);
+ WalSndWait(wakeEvents, sleeptime, wait_event);
}
/* reactivate latch so WalSndLoop knows to continue */
@@ -2265,6 +2358,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn)
{
ReplicationSlotMarkDirty();
ReplicationSlotsComputeRequiredLSN();
+ PhysicalWakeupLogicalWalSnd();
}
/*
@@ -3538,6 +3632,7 @@ WalSndShmemInit(void)
ConditionVariableInit(&WalSndCtl->wal_flush_cv);
ConditionVariableInit(&WalSndCtl->wal_replay_cv);
+ ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv);
}
}
@@ -3607,8 +3702,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event)
*
* And, we use separate shared memory CVs for physical and logical
* walsenders for selective wake ups, see WalSndWakeup() for more details.
+ *
+ * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV
+ * until awakened by physical walsenders after the walreceiver confirms
+ * the receipt of the LSN.
*/
- if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
+ if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+ else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv);
else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index ec2f31f82a..c08e00d1d6 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP
LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server."
LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server."
SSL_OPEN_SERVER "Waiting for SSL while attempting connection."
+WAIT_FOR_STANDBY_CONFIRMATION "Waiting for WAL to be received and flushed by the physical standby."
WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process."
WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 93ded31ed9..326cefcf4b 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4670,6 +4670,20 @@ struct config_string ConfigureNamesString[] =
check_debug_io_direct, assign_debug_io_direct, NULL
},
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("Lists streaming replication standby server slot "
+ "names that logical WAL sender processes will wait for."),
+ gettext_noop("Logical WAL sender processes will send decoded "
+ "changes to plugins only after the specified "
+ "replication slots confirm receiving WAL."),
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, assign_standby_slot_names, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index edcc0282b2..2244ee52f7 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -343,6 +343,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsender processes will wait for
# - Standby Servers -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index acbf567150..61a6b97952 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -226,6 +226,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
+extern PGDLLIMPORT char *standby_slot_names;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -274,4 +275,8 @@ extern void CheckSlotPermissions(void);
extern ReplicationSlotInvalidationCause
GetSlotInvalidationCause(const char *conflict_reason);
+extern List *GetStandbySlotList(void);
+extern bool StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel);
+extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn);
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index 0c3996e926..f2d8297f01 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -39,6 +39,7 @@ extern void InitWalSender(void);
extern bool exec_replication_command(const char *cmd_string);
extern void WalSndErrorCleanup(void);
extern void WalSndResourceCleanup(bool isCommit);
+extern void PhysicalWakeupLogicalWalSnd(void);
extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
extern void WalSndSignals(void);
extern Size WalSndShmemSize(void);
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 3113e9ea47..3c134c8edf 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -113,6 +113,13 @@ typedef struct
ConditionVariable wal_flush_cv;
ConditionVariable wal_replay_cv;
+ /*
+ * Used by physical walsenders holding slots specified in
+ * standby_slot_names to wake up logical walsenders holding failover
+ * enabled slots when a walreceiver confirms the receipt of LSN.
+ */
+ ConditionVariable wal_confirm_rcv_cv;
+
WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER];
} WalSndCtlData;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index c8a7aa9a11..d64dc5fcdb 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra,
extern void assign_wal_consistency_checking(const char *newval, void *extra);
extern bool check_wal_segment_size(int *newval, void **extra, GucSource source);
extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
+extern bool check_standby_slot_names(char **newval, void **extra,
+ GucSource source);
+extern void assign_standby_slot_names(const char *newval, void *extra);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl
index 5c7b4ca5e3..b95d95c06f 100644
--- a/src/test/recovery/t/006_logical_decoding.pl
+++ b/src/test/recovery/t/006_logical_decoding.pl
@@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'},
undef, 'logical slot was actually dropped with DB');
# Test logical slot advancing and its durability.
+# Passing failover=true (last arg) should not have any impact on advancing.
my $logical_slot = 'logical_slot';
$node_primary->safe_psql('postgres',
- "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);"
+ "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);"
);
$node_primary->psql(
'postgres', "
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 825c26da6f..9a43951b92 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -500,11 +500,213 @@ ok( $standby1->poll_query_until(
"SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot' AND synced AND NOT temporary;"),
'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby');
+##################################################
+# Test primary disallowing specified logical replication slots getting ahead of
+# specified physical replication slots. It uses the following set up:
+#
+# (physical standbys)
+# | ----> standby1 (primary_slot_name = sb1_slot)
+# | ----> standby2 (primary_slot_name = sb2_slot)
+# primary ----- |
+# (logical replication)
+# | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
+# | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
+#
+# standby_slot_names = 'sb1_slot'
+#
+# Set up is configured in such a way that the logical slot of subscriber1 is
+# enabled for failover, thus it will wait for the physical slot of
+# standby1(sb1_slot) to catch up before sending decoded changes to subscriber1.
+##################################################
+
+$backup_name = 'backup3';
+
+$primary->psql('postgres',
+ q{SELECT pg_create_physical_replication_slot('sb2_slot');});
+
+$primary->backup($backup_name);
+
+# Create another standby
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup(
+ $primary, $backup_name,
+ has_streaming => 1,
+ has_restoring => 1);
+$standby2->append_conf(
+ 'postgresql.conf', qq(
+primary_slot_name = 'sb2_slot'
+));
+$standby2->start;
+$primary->wait_for_replay_catchup($standby2);
+
+# Configure primary to disallow any logical slots that enabled failover from
+# getting ahead of specified physical replication slot (sb1_slot).
+$primary->append_conf(
+ 'postgresql.conf', qq(
+standby_slot_names = 'sb1_slot'
+));
+$primary->reload;
+
+# Create another subscriber node without enabling failover, wait for sync to
+# complete
+my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$subscriber2->init;
+$subscriber2->start;
+$subscriber2->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE tab_int (a int PRIMARY KEY);
+ CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot);
+]);
+
+$subscriber2->wait_for_subscription_sync;
+
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+# Stop the standby associated with the specified physical replication slot
+# (sb1_slot) so that the logical replication slot (lsub1_slot) won't receive
+# changes until the standby comes up.
+$standby1->stop;
+
+# Create some data on the primary
+my $primary_row_count = 20;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(11, $primary_row_count);");
+
+# Wait until the standby2 that's still running gets the data from the primary
+$primary->wait_for_replay_catchup($standby2);
+$result = $standby2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby2 gets data from primary");
+
+# Wait for regress_mysub2 to get the data from the primary. This subscription
+# was not enabled for failover so it gets the data without waiting for any
+# standbys.
+$primary->wait_for_catchup('regress_mysub2');
+$result = $subscriber2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "subscriber2 gets data from primary");
+
+# The regress_mysub1 was enabled for failover so it doesn't get the data from
+# primary and keeps waiting for the standby specified in standby_slot_names
+# (sb1_slot aka standby1).
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) <> $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data from primary until standby1 acknowledges changes"
+);
+
+# Start the standby specified in standby_slot_names (sb1_slot aka standby1) and
+# wait for it to catch up with the primary.
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+$result = $standby1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby1 gets data from primary");
+
+# Now that the standby specified in standby_slot_names is up and running, the
+# primary can send the decoded changes to the subscription enabled for failover
+# (i.e. regress_mysub1). While the standby was down, regress_mysub1 didn't
+# receive any data from the primary. i.e. the primary didn't allow it to go
+# ahead of standby.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 acknowledges changes");
+
+##################################################
+# Verify that when using pg_logical_slot_get_changes to consume changes from a
+# logical slot with failover enabled, it will also wait for the slots specified
+# in standby_slot_names to catch up.
+##################################################
+
+# Stop the standby associated with the specified physical replication slot so
+# that the logical replication slot won't receive changes until the standby
+# slot's restart_lsn is advanced or the slot is removed from the
+# standby_slot_names list.
+$primary->safe_psql('postgres', "TRUNCATE tab_int;");
+$primary->wait_for_catchup('regress_mysub1');
+$standby1->stop;
+
+# Create a logical 'test_decoding' replication slot with failover enabled
+$primary->safe_psql('postgres',
+ "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);"
+);
+
+my $back_q = $primary->background_psql(
+ 'postgres',
+ on_error_stop => 0,
+ timeout => $PostgreSQL::Test::Utils::timeout_default);
+
+# Try and get changes from the logical slot with failover enabled.
+my $offset = -s $primary->logfile;
+$back_q->query_until(qr//,
+ "SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);\n");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the function
+# pg_logical_slot_get_changes should now return, and the session can be
+# stopped.
+$back_q->quit;
+
+$primary->safe_psql('postgres',
+ "SELECT pg_drop_replication_slot('test_slot');"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
+##################################################
+# Test that logical replication will wait for the user-created inactive
+# physical slot to catch up until we remove the slot from standby_slot_names.
+##################################################
+
+# Create some data on the primary
+$primary_row_count = 10;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);");
+
+# The regress_mysub1 doesn't get the data from primary because the specified
+# standby slot (sb1_slot) in standby_slot_names is inactive.
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data as the sb1_slot doesn't catch up");
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the primary server should now
+# send the decoded changes to the subscription.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list"
+);
+
##################################################
# Promote the standby1 to primary. Confirm that:
# a) the slot 'lsub1_slot' is retained on the new primary
# b) logical replication for regress_mysub1 is resumed successfully after failover
##################################################
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+
$standby1->promote;
# Update subscription with the new primary's connection info
--
2.30.0.windows.2
[application/octet-stream] v102_2-0002-Document-the-steps-to-check-if-the-standby-is-r.patch (7.0K, ../../OS0PR01MB571669D7F110110F7D87C48D945E2@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v102_2-0002-Document-the-steps-to-check-if-the-standby-is-r.patch)
download | inline diff:
From d07ca9bc9528a90a7b5a6ce155b350abe177ae21 Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Fri, 19 Jan 2024 11:04:16 +0530
Subject: [PATCH v102 2/2] Document the steps to check if the standby is ready
for failover
---
doc/src/sgml/high-availability.sgml | 9 ++
doc/src/sgml/logical-replication.sgml | 136 ++++++++++++++++++++++++++
2 files changed, 145 insertions(+)
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 236c0af65f..36215aa68c 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -1487,6 +1487,15 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)'
Written administration procedures are advised.
</para>
+ <para>
+ If you have opted for synchronization of logical slots (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>),
+ then before switching to the standby server, it is recommended to check
+ if the logical slots synchronized on the standby server are ready
+ for failover. This can be done by following the steps described in
+ <xref linkend="logical-replication-failover"/>.
+ </para>
+
<para>
To trigger failover of a log-shipping standby server, run
<command>pg_ctl promote</command> or call <function>pg_promote()</function>.
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ec2130669e..be59d306a1 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -687,6 +687,142 @@ ALTER SUBSCRIPTION
</sect1>
+ <sect1 id="logical-replication-failover">
+ <title>Logical Replication Failover</title>
+
+ <para>
+ When the publisher server is the primary server of a streaming replication,
+ the logical slots on that primary server can be synchronized to the standby
+ server by specifying <literal>failover = true</literal> when creating
+ subscriptions for those publications. Enabling failover ensures a seamless
+ transition of those subscriptions after the standby is promoted. They can
+ continue subscribing to publications now on the new primary server without
+ any data loss.
+ </para>
+
+ <para>
+ Because the slot synchronization logic copies asynchronously, it is
+ necessary to confirm that replication slots have been synced to the standby
+ server before the failover happens. Furthermore, to ensure a successful
+ failover, the standby server must not be lagging behind the subscriber. It
+ is highly recommended to use
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ to prevent the subscriber from consuming changes faster than the hot standby.
+ To confirm that the standby server is indeed ready for failover, follow
+ these 2 steps:
+ </para>
+
+ <procedure>
+ <step performance="required">
+ <para>
+ Confirm that all the necessary logical replication slots have been synced to
+ the standby server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node, use the following SQL to identify
+ which slots should be synced to the standby that we plan to promote.
+<programlisting>
+test_sub=# SELECT
+ array_agg(slotname) AS slots
+ FROM
+ ((
+ SELECT r.srsubid AS subid, CONCAT('pg_', srsubid, '_sync_', srrelid, '_', ctl.system_identifier) AS slotname
+ FROM pg_control_system() ctl, pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate = 'f' AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT s.oid AS subid, s.subslotname as slotname
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ slots
+-------
+ {sub1,sub2,sub3}
+(1 row)
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, check that the logical replication slots identified above exist on
+ the standby server and are ready for failover.
+<programlisting>
+test_standby=# SELECT slot_name, (synced AND NOT temporary AND conflict_reason IS NULL) AS failover_ready
+ FROM pg_replication_slots
+ WHERE slot_name IN ('sub1','sub2','sub3');
+ slot_name | failover_ready
+-------------+----------------
+ sub1 | t
+ sub2 | t
+ sub3 | t
+(3 rows)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+
+ <step performance="required">
+ <para>
+ Confirm that the standby server is not lagging behind the subscribers.
+ This step can be skipped if
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ has been correctly configured. If standby_slot_names is not configured
+ correctly, it is highly recommended to run this step after the primary
+ server is down, otherwise the results of the query may vary at different
+ points of time due to the ongoing replication on the logical subscribers
+ from the primary server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node check the last replayed WAL.
+ This step needs to be run on the database(s) that includes the failover
+ enabled subscription(s), to find the last replayed WAL on each database.
+<programlisting>
+test_sub=# SELECT
+ MAX(remote_lsn) AS remote_lsn_on_subscriber
+ FROM
+ ((
+ SELECT (CASE WHEN r.srsubstate = 'f' THEN pg_replication_origin_progress(CONCAT('pg_', r.srsubid, '_', r.srrelid), false)
+ WHEN r.srsubstate IN ('s', 'r') THEN r.srsublsn END) AS remote_lsn
+ FROM pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate IN ('f', 's', 'r') AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT pg_replication_origin_progress(CONCAT('pg_', s.oid), false) AS remote_lsn
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ remote_lsn_on_subscriber
+--------------------------
+ 0/3000388
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, on the standby server check that the last-received WAL location
+ is ahead of the replayed WAL location(s) on the subscriber identified
+ above. If the above SQL result was NULL, it means the subscriber has not
+ yet replayed any WAL, so the standby server must be ahead of the
+ subscriber, and this step can be skipped.
+<programlisting>
+test_standby=# SELECT pg_last_wal_receive_lsn() >= '0/3000388'::pg_lsn AS failover_ready;
+ failover_ready
+----------------
+ t
+(1 row)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+ </procedure>
+
+ <para>
+ If the result (<literal>failover_ready</literal>) of both above steps is
+ true, existing subscriptions will be able to continue without data loss.
+ </para>
+
+ </sect1>
+
<sect1 id="logical-replication-row-filter">
<title>Row Filters</title>
--
2.30.0.windows.2
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-01 04:22 Peter Smith <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
1 sibling, 3 replies; 68+ messages in thread
From: Peter Smith @ 2024-03-01 04:22 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
Here are some review comments for v102-0001.
======
doc/src/sgml/config.sgml
1.
+ <para>
+ Lists the streaming replication standby server slot names that logical
+ WAL sender processes will wait for. Logical WAL sender processes will
+ send decoded changes to plugins only after the specified replication
+ slots confirm receiving WAL. This guarantees that logical replication
+ slots with failover enabled do not consume changes until those changes
+ are received and flushed to corresponding physical standbys. If a
+ logical replication connection is meant to switch to a physical standby
+ after the standby is promoted, the physical replication slot for the
+ standby should be listed here. Note that logical replication will not
+ proceed if the slots specified in the standby_slot_names do
not exist or
+ are invalidated.
+ </para>
Should this also mention the effect this GUC has on those 2 SQL
functions? E.g. Commit message says:
Additionally, The SQL functions pg_logical_slot_get_changes and
pg_replication_slot_advance are modified to wait for the replication
slots mentioned in 'standby_slot_names' to catch up before returning.
======
src/backend/replication/slot.c
2. validate_standby_slots
+ else if (!ReplicationSlotCtl)
+ {
+ /*
+ * We cannot validate the replication slot if the replication slots'
+ * data has not been initialized. This is ok as we will validate the
+ * specified slot when waiting for them to catch up. See
+ * StandbySlotsHaveCaughtup for details.
+ */
+ }
+ else
+ {
+ /*
+ * If the replication slots' data have been initialized, verify if the
+ * specified slots exist and are logical slots.
+ */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
IMO that 2nd comment does not need to say "If the replication slots'
data have been initialized," because that is implicit from the
if/else.
~~~
3. GetStandbySlotList
+List *
+GetStandbySlotList(void)
+{
+ if (RecoveryInProgress())
+ return NIL;
+ else
+ return standby_slot_names_list;
+}
The 'else' is not needed. IMO is better without it, but YMMV.
~~~
4. StandbySlotsHaveCaughtup
+/*
+ * Return true if the slots specified in standby_slot_names have caught up to
+ * the given WAL location, false otherwise.
+ *
+ * The elevel parameter determines the error level used for logging messages
+ * related to slots that do not exist, are invalidated, or are inactive.
+ */
+bool
+StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel)
/determines/specifies/
~
5.
+ /*
+ * Don't need to wait for the standby to catch up if there is no value in
+ * standby_slot_names.
+ */
+ if (!standby_slot_names_list)
+ return true;
+
+ /*
+ * If we are on a standby server, we do not need to wait for the standby to
+ * catch up since we do not support syncing slots to cascading standbys.
+ */
+ if (RecoveryInProgress())
+ return true;
+
+ /*
+ * Return true if all the standbys have already caught up to the passed in
+ * WAL localtion.
+ */
+ if (!XLogRecPtrIsInvalid(standby_slot_oldest_flush_lsn) &&
+ standby_slot_oldest_flush_lsn >= wait_for_lsn)
+ return true;
5a.
I felt all these comments should be worded in a consistent way like
"Don't need to wait ..."
e.g.
1. Don't need to wait for the standbys to catch up if there is no
value in 'standby_slot_names'.
2. Don't need to wait for the standbys to catch up if we are on a
standby server, since we do not support syncing slots to cascading
standbys.
3. Don't need to wait for the standbys to catch up if they are already
beyond the specified WAL location.
~
5b.
typo
/WAL localtion/WAL location/
~~~
6.
+ if (!slot)
+ {
+ /*
+ * It may happen that the slot specified in standby_slot_names GUC
+ * value is dropped, so let's skip over it.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not exist",
+ name, "standby_slot_names"));
+ continue;
+ }
Is "is dropped" strictly the only reason? IIUC another reason is the
slot maybe just did not even exist in the first place but it was not
detected before now because inititial validation was skipped.
~~~
7.
+ /*
+ * Return false if not all the standbys have caught up to the specified WAL
+ * location.
+ */
+ if (caught_up_slot_num != list_length(standby_slot_names_list))
+ return false;
Somehow it seems complicated to have a counter of the slots as you
process then compare that counter to the list_length to determine if
one of them was skipped.
Probably simpler just to set a 'skipped' flag set whenever you do 'continue'...
======
src/backend/replication/walsender.c
8.
+/*
+ * Returns true if there are not enough WALs to be read, or if not all standbys
+ * have caught up to the flushed position when failover_slot is true;
+ * otherwise, returns false.
+ *
+ * Set prioritize_stop to true to skip waiting for WALs if the shutdown signal
+ * is received.
+ *
+ * Set failover_slot to true if the current acquired slot is a failover enabled
+ * slot and we are streaming.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ bool prioritize_stop, bool failover_slot,
+ uint32 *wait_event)
8a.
/Set prioritize_stop to true/Specify prioritize_stop=true/
/Set failover_slot to true/Specify failover_slot=true/
~
8b.
Aren't the static function names typically snake_case?
~~~
9.
+ /*
+ * Check if we need to wait for WALs to be flushed to disk. We don't need
+ * to wait for WALs after receiving the shutdown signal unless the
+ * wait_for_wal_on_stop is true.
+ */
+ if (target_lsn > flushed_lsn && !(prioritize_stop && got_STOPPING))
+ *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
The comment says 'wait_for_wal_on_stop' but the code says 'prioritize_stop' (??)
~~~
10.
+ /*
+ * Check if we need to wait for WALs to be flushed to disk. We don't need
+ * to wait for WALs after receiving the shutdown signal unless the
+ * wait_for_wal_on_stop is true.
+ */
+ if (target_lsn > flushed_lsn && !(prioritize_stop && got_STOPPING))
+ *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
+
+ /*
+ * Check if the standby slots have caught up to the flushed position. It is
+ * good to wait up to RecentFlushPtr and then let it send the changes to
+ * logical subscribers one by one which are already covered in
+ * RecentFlushPtr without needing to wait on every change for standby
+ * confirmation. Note that after receiving the shutdown signal, an ERROR is
+ * reported if any slots are dropped, invalidated, or inactive. This
+ * measure is taken to prevent the walsender from waiting indefinitely.
+ */
+ else if (failover_slot && !StandbySlotsHaveCaughtup(flushed_lsn, elevel))
+ *wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
+ else
+ return false;
+
+ return true;
This if/else/else seems overly difficult to read. IMO it will be
easier if written like:
SUGGESTION
<comment>
if (target_lsn > flushed_lsn && !(prioritize_stop && got_STOPPING))
{
*wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
return true;
}
<comment>
if (failover_slot && !StandbySlotsHaveCaughtup(flushed_lsn, elevel))
{
*wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
return true;
}
return false;
----------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-01 06:10 Masahiko Sawada <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 3 replies; 68+ messages in thread
From: Masahiko Sawada @ 2024-03-01 06:10 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Fri, Mar 1, 2024 at 12:42 PM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> On Friday, March 1, 2024 10:17 AM Zhijie Hou (Fujitsu) <[email protected]> wrote:
> >
> >
> > Attach the V102 patch set which addressed Amit and Shveta's comments.
> > Thanks Shveta for helping addressing the comments off-list.
>
> The cfbot reported a compile warning, here is the new version patch which fixed it,
> Also removed some outdate comments in this version.
>
Thank you for updating the patch!
I've reviewed the v102-0001 patch. Here are some comments:
---
I got a compiler warning:
walsender.c:1829:6: warning: variable 'wait_event' is used
uninitialized whenever '&&' condition is false
[-Wsometimes-uninitialized]
if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
walsender.c:1871:7: note: uninitialized use occurs here
if (wait_event == WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL)
^~~~~~~~~~
walsender.c:1829:6: note: remove the '&&' if its condition is always true
if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
walsender.c:1818:20: note: initialize the variable 'wait_event' to
silence this warning
uint32 wait_event;
^
= 0
1 warning generated.
---
+void
+assign_standby_slot_names(const char *newval, void *extra)
+{
+ List *standby_slots;
+ MemoryContext oldcxt;
+ char *standby_slot_names_cpy = extra;
+
Given that the newval and extra have the same data (standby_slot_names
value), why do we not use newval instead? I think that if we use
newval, we don't need to guc_strdup() in check_standby_slot_names(),
we might need to do list_copy_deep() instead, though. It's not clear
to me as there is no comment.
---
+
+ standby_slot_oldest_flush_lsn = min_restart_lsn;
+
IIUC we expect that standby_slot_oldest_flush_lsn never moves
backward. If so, I think it's better to have an assertion here.
---
Resetting standby_slot_names doesn't work:
=# alter system set standby_slot_names to '';
ERROR: invalid value for parameter "standby_slot_names": """"
DETAIL: replication slot "" does not exist
---
+ /*
+ * Switch to the memory context under which GUC variables are allocated
+ * (GUCMemoryContext).
+ */
+ oldcxt =
MemoryContextSwitchTo(GetMemoryChunkContext(standby_slot_names_cpy));
+ standby_slot_names_list = list_copy(standby_slots);
+ MemoryContextSwitchTo(oldcxt);
Why do we not explicitly switch to GUCMemoryContext?
---
+ if (!standby_slot_names_list)
+ return true;
+
Probably "standby_slot_names_list == NIL" is more consistent with
other places. The same can be applied in WaitForStandbyConfirmation().
---
+ /*
+ * Return true if all the standbys have already caught up to
the passed in
+ * WAL localtion.
+ */
+
s/localtion/location/
---
I was a bit surprised by the fact that standby_slot_names value is
handled in a different way than a similar parameter
synchronous_standby_names. For example, the following command doesn't
work unless there is a replication slot 'slot1, slot2':
=# alter system set standby_slot_names to 'slot1, slot2';
ERROR: invalid value for parameter "standby_slot_names": ""slot1, slot2""
DETAIL: replication slot "slot1, slot2" does not exist
Whereas "alter system set synchronous_standby_names to stb1, stb2;"
can correctly separate the string into 'stb1' and 'stb2'.
Probably it would be okay since this behavior of standby_slot_names is
the same as other GUC parameters that accept a comma-separated string.
But I was confused a bit the first time I used it.
---
+ /*
+ * "*" is not accepted as in that case primary will not be able to know
+ * for which all standbys to wait for. Even if we have physical slots
+ * info, there is no way to confirm whether there is any standby
+ * configured for the known physical slots.
+ */
+ if (strcmp(*newval, "*") == 0)
+ {
+ GUC_check_errdetail("\"*\" is not accepted for
standby_slot_names");
+ return false;
+ }
Why only '*' is checked aside from validate_standby_slots()? I think
that the doc doesn't mention anything about '*' and '*' cannot be used
as a replication slot name. So even if we don't have this check, it
might be no problem.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-01 06:21 Peter Smith <[email protected]>
parent: Masahiko Sawada <[email protected]>
2 siblings, 1 reply; 68+ messages in thread
From: Peter Smith @ 2024-03-01 06:21 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; Amit Kapila <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Fri, Mar 1, 2024 at 5:11 PM Masahiko Sawada <[email protected]> wrote:
>
...
> + /*
> + * "*" is not accepted as in that case primary will not be able to know
> + * for which all standbys to wait for. Even if we have physical slots
> + * info, there is no way to confirm whether there is any standby
> + * configured for the known physical slots.
> + */
> + if (strcmp(*newval, "*") == 0)
> + {
> + GUC_check_errdetail("\"*\" is not accepted for
> standby_slot_names");
> + return false;
> + }
>
> Why only '*' is checked aside from validate_standby_slots()? I think
> that the doc doesn't mention anything about '*' and '*' cannot be used
> as a replication slot name. So even if we don't have this check, it
> might be no problem.
>
Hi, a while ago I asked this same question. See [1 #28] for the response..
----------
[1] https://www.postgresql.org/message-id/OS0PR01MB571646B8186F6A06404BD50194BDA%40OS0PR01MB5716.jpnprd0...
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-01 06:45 Amit Kapila <[email protected]>
parent: Peter Smith <[email protected]>
2 siblings, 0 replies; 68+ messages in thread
From: Amit Kapila @ 2024-03-01 06:45 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Fri, Mar 1, 2024 at 9:53 AM Peter Smith <[email protected]> wrote:
>
> Here are some review comments for v102-0001.
>
>
> 7.
> + /*
> + * Return false if not all the standbys have caught up to the specified WAL
> + * location.
> + */
> + if (caught_up_slot_num != list_length(standby_slot_names_list))
> + return false;
>
> Somehow it seems complicated to have a counter of the slots as you
> process then compare that counter to the list_length to determine if
> one of them was skipped.
>
> Probably simpler just to set a 'skipped' flag set whenever you do 'continue'...
>
The other thing is do we need to continue when we find some slot that
can't be processed? If not, then we can simply set the boolean flag,
break the loop, and return false if the boolean is set after releasing
the LWLock. The other way is we simply release lock whenever we need
to skip the slot and just return false.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-01 07:15 Bertrand Drouvot <[email protected]>
parent: Peter Smith <[email protected]>
2 siblings, 0 replies; 68+ messages in thread
From: Bertrand Drouvot @ 2024-03-01 07:15 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
Hi,
On Fri, Mar 01, 2024 at 03:22:55PM +1100, Peter Smith wrote:
> Here are some review comments for v102-0001.
>
> ======
> doc/src/sgml/config.sgml
>
> 1.
> + <para>
> + Lists the streaming replication standby server slot names that logical
> + WAL sender processes will wait for. Logical WAL sender processes will
> + send decoded changes to plugins only after the specified replication
> + slots confirm receiving WAL. This guarantees that logical replication
> + slots with failover enabled do not consume changes until those changes
> + are received and flushed to corresponding physical standbys. If a
> + logical replication connection is meant to switch to a physical standby
> + after the standby is promoted, the physical replication slot for the
> + standby should be listed here. Note that logical replication will not
> + proceed if the slots specified in the standby_slot_names do
> not exist or
> + are invalidated.
> + </para>
>
> Should this also mention the effect this GUC has on those 2 SQL
> functions? E.g. Commit message says:
>
> Additionally, The SQL functions pg_logical_slot_get_changes and
> pg_replication_slot_advance are modified to wait for the replication
> slots mentioned in 'standby_slot_names' to catch up before returning.
I think that's also true for all the ones that rely on
pg_logical_slot_get_changes_guts(), means:
- pg_logical_slot_get_changes
- pg_logical_slot_peek_changes
- pg_logical_slot_get_binary_changes
- pg_logical_slot_peek_binary_changes
Not sure it's worth to mention the "binary" ones though as their doc mention
they behave as their "non binary" counterpart.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-01 07:21 Zhijie Hou (Fujitsu) <[email protected]>
parent: Masahiko Sawada <[email protected]>
2 siblings, 1 reply; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-01 07:21 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Friday, March 1, 2024 2:11 PM Masahiko Sawada <[email protected]> wrote:
>
> On Fri, Mar 1, 2024 at 12:42 PM Zhijie Hou (Fujitsu) <[email protected]>
> wrote:
> >
> > On Friday, March 1, 2024 10:17 AM Zhijie Hou (Fujitsu)
> <[email protected]> wrote:
> > >
> > >
> > > Attach the V102 patch set which addressed Amit and Shveta's comments.
> > > Thanks Shveta for helping addressing the comments off-list.
> >
> > The cfbot reported a compile warning, here is the new version patch
> > which fixed it, Also removed some outdate comments in this version.
> >
>
> I've reviewed the v102-0001 patch. Here are some comments:
Thanks for the comments !
>
> ---
> I got a compiler warning:
>
> walsender.c:1829:6: warning: variable 'wait_event' is used uninitialized
> whenever '&&' condition is false [-Wsometimes-uninitialized]
> if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> walsender.c:1871:7: note: uninitialized use occurs here
> if (wait_event ==
> WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL)
> ^~~~~~~~~~
> walsender.c:1829:6: note: remove the '&&' if its condition is always true
> if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> walsender.c:1818:20: note: initialize the variable 'wait_event' to silence this
> warning
> uint32 wait_event;
> ^
> = 0
> 1 warning generated.
Thanks for reporting, it was fixed in V102_2.
>
> ---
> +void
> +assign_standby_slot_names(const char *newval, void *extra) {
> + List *standby_slots;
> + MemoryContext oldcxt;
> + char *standby_slot_names_cpy = extra;
> +
>
> Given that the newval and extra have the same data (standby_slot_names
> value), why do we not use newval instead? I think that if we use
> newval, we don't need to guc_strdup() in check_standby_slot_names(),
> we might need to do list_copy_deep() instead, though. It's not clear
> to me as there is no comment.
I think SplitIdentifierString will modify the passed in string, so we'd better
not pass the newval to it, otherwise the stored guc string(standby_slot_names)
will be changed. I can see we are doing similar thing in other GUC check/assign
function as well. (check_wal_consistency_checking/
assign_wal_consistency_checking, check_createrole_self_grant/
assign_createrole_self_grant ...).
> ---
> + /*
> + * Switch to the memory context under which GUC variables are
> allocated
> + * (GUCMemoryContext).
> + */
> + oldcxt =
> MemoryContextSwitchTo(GetMemoryChunkContext(standby_slot_names_cpy
> ));
> + standby_slot_names_list = list_copy(standby_slots);
> + MemoryContextSwitchTo(oldcxt);
>
> Why do we not explicitly switch to GUCMemoryContext?
I think it's because the GUCMemoryContext is not exposed.
Best Regards,
Hou zj
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-01 09:21 Amit Kapila <[email protected]>
parent: Masahiko Sawada <[email protected]>
2 siblings, 0 replies; 68+ messages in thread
From: Amit Kapila @ 2024-03-01 09:21 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Fri, Mar 1, 2024 at 11:41 AM Masahiko Sawada <[email protected]> wrote:
>
> On Fri, Mar 1, 2024 at 12:42 PM Zhijie Hou (Fujitsu)
> <[email protected]> wrote:
> ---
> I was a bit surprised by the fact that standby_slot_names value is
> handled in a different way than a similar parameter
> synchronous_standby_names. For example, the following command doesn't
> work unless there is a replication slot 'slot1, slot2':
>
> =# alter system set standby_slot_names to 'slot1, slot2';
> ERROR: invalid value for parameter "standby_slot_names": ""slot1, slot2""
> DETAIL: replication slot "slot1, slot2" does not exist
>
> Whereas "alter system set synchronous_standby_names to stb1, stb2;"
> can correctly separate the string into 'stb1' and 'stb2'.
>
> Probably it would be okay since this behavior of standby_slot_names is
> the same as other GUC parameters that accept a comma-separated string.
> But I was confused a bit the first time I used it.
>
I think it is better to keep the behavior in this regard the same as
'synchronous_standby_names' because both have similarities w.r.t
replication.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-02 03:51 Zhijie Hou (Fujitsu) <[email protected]>
parent: Peter Smith <[email protected]>
2 siblings, 2 replies; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-02 03:51 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Friday, March 1, 2024 12:23 PM Peter Smith <[email protected]> wrote:
>
> Here are some review comments for v102-0001.
>
> ======
> doc/src/sgml/config.sgml
>
> 1.
> + <para>
> + Lists the streaming replication standby server slot names that logical
> + WAL sender processes will wait for. Logical WAL sender processes will
> + send decoded changes to plugins only after the specified replication
> + slots confirm receiving WAL. This guarantees that logical replication
> + slots with failover enabled do not consume changes until those
> changes
> + are received and flushed to corresponding physical standbys. If a
> + logical replication connection is meant to switch to a physical standby
> + after the standby is promoted, the physical replication slot for the
> + standby should be listed here. Note that logical replication will not
> + proceed if the slots specified in the standby_slot_names do
> not exist or
> + are invalidated.
> + </para>
>
> Should this also mention the effect this GUC has on those 2 SQL functions? E.g.
> Commit message says:
>
> Additionally, The SQL functions pg_logical_slot_get_changes and
> pg_replication_slot_advance are modified to wait for the replication slots
> mentioned in 'standby_slot_names' to catch up before returning.
Added.
>
> ======
> src/backend/replication/slot.c
>
> 2. validate_standby_slots
>
> + else if (!ReplicationSlotCtl)
> + {
> + /*
> + * We cannot validate the replication slot if the replication slots'
> + * data has not been initialized. This is ok as we will validate the
> + * specified slot when waiting for them to catch up. See
> + * StandbySlotsHaveCaughtup for details.
> + */
> + }
> + else
> + {
> + /*
> + * If the replication slots' data have been initialized, verify if the
> + * specified slots exist and are logical slots.
> + */
> + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
>
> IMO that 2nd comment does not need to say "If the replication slots'
> data have been initialized," because that is implicit from the if/else.
Removed.
>
> ~~~
>
> 3. GetStandbySlotList
>
> +List *
> +GetStandbySlotList(void)
> +{
> + if (RecoveryInProgress())
> + return NIL;
> + else
> + return standby_slot_names_list;
> +}
>
> The 'else' is not needed. IMO is better without it, but YMMV.
Removed.
>
> ~~~
>
> 4. StandbySlotsHaveCaughtup
>
> +/*
> + * Return true if the slots specified in standby_slot_names have caught
> +up to
> + * the given WAL location, false otherwise.
> + *
> + * The elevel parameter determines the error level used for logging
> +messages
> + * related to slots that do not exist, are invalidated, or are inactive.
> + */
> +bool
> +StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel)
>
> /determines/specifies/
Changed.
>
> ~
>
> 5.
> + /*
> + * Don't need to wait for the standby to catch up if there is no value
> + in
> + * standby_slot_names.
> + */
> + if (!standby_slot_names_list)
> + return true;
> +
> + /*
> + * If we are on a standby server, we do not need to wait for the
> + standby to
> + * catch up since we do not support syncing slots to cascading standbys.
> + */
> + if (RecoveryInProgress())
> + return true;
> +
> + /*
> + * Return true if all the standbys have already caught up to the passed
> + in
> + * WAL localtion.
> + */
> + if (!XLogRecPtrIsInvalid(standby_slot_oldest_flush_lsn) &&
> + standby_slot_oldest_flush_lsn >= wait_for_lsn) return true;
>
>
> 5a.
> I felt all these comments should be worded in a consistent way like "Don't need
> to wait ..."
>
> e.g.
> 1. Don't need to wait for the standbys to catch up if there is no value in
> 'standby_slot_names'.
> 2. Don't need to wait for the standbys to catch up if we are on a standby server,
> since we do not support syncing slots to cascading standbys.
> 3. Don't need to wait for the standbys to catch up if they are already beyond
> the specified WAL location.
Changed.
>
> ~
>
> 5b.
> typo
> /WAL localtion/WAL location/
>
Fixed.
> ~~~
>
> 6.
> + if (!slot)
> + {
> + /*
> + * It may happen that the slot specified in standby_slot_names GUC
> + * value is dropped, so let's skip over it.
> + */
> + ereport(elevel,
> + errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> + errmsg("replication slot \"%s\" specified in parameter %s does not exist",
> + name, "standby_slot_names"));
> + continue;
> + }
>
> Is "is dropped" strictly the only reason? IIUC another reason is the slot maybe
> just did not even exist in the first place but it was not detected before now
> because inititial validation was skipped.
Changed the comment.
>
> ~~~
>
> 7.
> + /*
> + * Return false if not all the standbys have caught up to the specified
> + WAL
> + * location.
> + */
> + if (caught_up_slot_num != list_length(standby_slot_names_list))
> + return false;
>
> Somehow it seems complicated to have a counter of the slots as you process
> then compare that counter to the list_length to determine if one of them was
> skipped.
>
> Probably simpler just to set a 'skipped' flag set whenever you do 'continue'...
>
I prefer the current style because we need to set skipped =true in
multiple places which doesn't seem better to me.
> ======
> src/backend/replication/walsender.c
>
> 8.
> +/*
> + * Returns true if there are not enough WALs to be read, or if not all
> +standbys
> + * have caught up to the flushed position when failover_slot is true;
> + * otherwise, returns false.
> + *
> + * Set prioritize_stop to true to skip waiting for WALs if the shutdown
> +signal
> + * is received.
> + *
> + * Set failover_slot to true if the current acquired slot is a failover
> +enabled
> + * slot and we are streaming.
> + *
> + * If returning true, the function sets the appropriate wait event in
> + * wait_event; otherwise, wait_event is set to 0.
> + */
> +static bool
> +NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn, bool
> +prioritize_stop, bool failover_slot,
> + uint32 *wait_event)
>
> 8a.
> /Set prioritize_stop to true/Specify prioritize_stop=true/
>
> /Set failover_slot to true/Specify failover_slot=true/
This function has been refactored a bit.
>
> ~
>
> 8b.
> Aren't the static function names typically snake_case?
I think the current name style is more consistent with the other functions in walsender.c.
>
> ~~~
>
> 9.
> + /*
> + * Check if we need to wait for WALs to be flushed to disk. We don't
> + need
> + * to wait for WALs after receiving the shutdown signal unless the
> + * wait_for_wal_on_stop is true.
> + */
> + if (target_lsn > flushed_lsn && !(prioritize_stop && got_STOPPING))
> + *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
>
> The comment says 'wait_for_wal_on_stop' but the code says 'prioritize_stop'
> (??)
This has been removed in last version.
> ~~~
>
> 10.
> + /*
> + * Check if we need to wait for WALs to be flushed to disk. We don't
> + need
> + * to wait for WALs after receiving the shutdown signal unless the
> + * wait_for_wal_on_stop is true.
> + */
> + if (target_lsn > flushed_lsn && !(prioritize_stop && got_STOPPING))
> + *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
> +
> + /*
> + * Check if the standby slots have caught up to the flushed position.
> + It is
> + * good to wait up to RecentFlushPtr and then let it send the changes
> + to
> + * logical subscribers one by one which are already covered in
> + * RecentFlushPtr without needing to wait on every change for standby
> + * confirmation. Note that after receiving the shutdown signal, an
> + ERROR is
> + * reported if any slots are dropped, invalidated, or inactive. This
> + * measure is taken to prevent the walsender from waiting indefinitely.
> + */
> + else if (failover_slot && !StandbySlotsHaveCaughtup(flushed_lsn,
> + elevel)) *wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
> + else
> + return false;
> +
> + return true;
>
> This if/else/else seems overly difficult to read. IMO it will be easier if written
> like:
>
> SUGGESTION
> <comment>
> if (target_lsn > flushed_lsn && !(prioritize_stop && got_STOPPING)) {
> *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
> return true;
> }
>
> <comment>
> if (failover_slot && !StandbySlotsHaveCaughtup(flushed_lsn, elevel)) {
> *wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
> return true;
> }
>
> return false;
Changed.
Attach the V103 patch set which addressed above comments and
Sawada-san's comment[1].
Apart from the comments, the code in WalSndWaitForWal was refactored
a bit to make it neater. Thanks Shveta for helping writing the code and doc.
[1] https://www.postgresql.org/message-id/CAD21AoBhty79zHgXYMNHH1KqO2VtmjRi22QPmYP2yaHC9WFVdw%40mail.gma...
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v103-0002-Document-the-steps-to-check-if-the-standby-is-r.patch (7.0K, ../../OS0PR01MB5716B48653125E199A0901D3945D2@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v103-0002-Document-the-steps-to-check-if-the-standby-is-r.patch)
download | inline diff:
From 51a7390aac9ae9b5c85193ee69d143f38dc10b64 Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Fri, 19 Jan 2024 11:04:16 +0530
Subject: [PATCH v103 2/2] Document the steps to check if the standby is ready
for failover
---
doc/src/sgml/high-availability.sgml | 9 ++
doc/src/sgml/logical-replication.sgml | 136 ++++++++++++++++++++++++++
2 files changed, 145 insertions(+)
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 236c0af65f..36215aa68c 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -1487,6 +1487,15 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)'
Written administration procedures are advised.
</para>
+ <para>
+ If you have opted for synchronization of logical slots (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>),
+ then before switching to the standby server, it is recommended to check
+ if the logical slots synchronized on the standby server are ready
+ for failover. This can be done by following the steps described in
+ <xref linkend="logical-replication-failover"/>.
+ </para>
+
<para>
To trigger failover of a log-shipping standby server, run
<command>pg_ctl promote</command> or call <function>pg_promote()</function>.
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ec2130669e..be59d306a1 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -687,6 +687,142 @@ ALTER SUBSCRIPTION
</sect1>
+ <sect1 id="logical-replication-failover">
+ <title>Logical Replication Failover</title>
+
+ <para>
+ When the publisher server is the primary server of a streaming replication,
+ the logical slots on that primary server can be synchronized to the standby
+ server by specifying <literal>failover = true</literal> when creating
+ subscriptions for those publications. Enabling failover ensures a seamless
+ transition of those subscriptions after the standby is promoted. They can
+ continue subscribing to publications now on the new primary server without
+ any data loss.
+ </para>
+
+ <para>
+ Because the slot synchronization logic copies asynchronously, it is
+ necessary to confirm that replication slots have been synced to the standby
+ server before the failover happens. Furthermore, to ensure a successful
+ failover, the standby server must not be lagging behind the subscriber. It
+ is highly recommended to use
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ to prevent the subscriber from consuming changes faster than the hot standby.
+ To confirm that the standby server is indeed ready for failover, follow
+ these 2 steps:
+ </para>
+
+ <procedure>
+ <step performance="required">
+ <para>
+ Confirm that all the necessary logical replication slots have been synced to
+ the standby server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node, use the following SQL to identify
+ which slots should be synced to the standby that we plan to promote.
+<programlisting>
+test_sub=# SELECT
+ array_agg(slotname) AS slots
+ FROM
+ ((
+ SELECT r.srsubid AS subid, CONCAT('pg_', srsubid, '_sync_', srrelid, '_', ctl.system_identifier) AS slotname
+ FROM pg_control_system() ctl, pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate = 'f' AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT s.oid AS subid, s.subslotname as slotname
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ slots
+-------
+ {sub1,sub2,sub3}
+(1 row)
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, check that the logical replication slots identified above exist on
+ the standby server and are ready for failover.
+<programlisting>
+test_standby=# SELECT slot_name, (synced AND NOT temporary AND conflict_reason IS NULL) AS failover_ready
+ FROM pg_replication_slots
+ WHERE slot_name IN ('sub1','sub2','sub3');
+ slot_name | failover_ready
+-------------+----------------
+ sub1 | t
+ sub2 | t
+ sub3 | t
+(3 rows)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+
+ <step performance="required">
+ <para>
+ Confirm that the standby server is not lagging behind the subscribers.
+ This step can be skipped if
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ has been correctly configured. If standby_slot_names is not configured
+ correctly, it is highly recommended to run this step after the primary
+ server is down, otherwise the results of the query may vary at different
+ points of time due to the ongoing replication on the logical subscribers
+ from the primary server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node check the last replayed WAL.
+ This step needs to be run on the database(s) that includes the failover
+ enabled subscription(s), to find the last replayed WAL on each database.
+<programlisting>
+test_sub=# SELECT
+ MAX(remote_lsn) AS remote_lsn_on_subscriber
+ FROM
+ ((
+ SELECT (CASE WHEN r.srsubstate = 'f' THEN pg_replication_origin_progress(CONCAT('pg_', r.srsubid, '_', r.srrelid), false)
+ WHEN r.srsubstate IN ('s', 'r') THEN r.srsublsn END) AS remote_lsn
+ FROM pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate IN ('f', 's', 'r') AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT pg_replication_origin_progress(CONCAT('pg_', s.oid), false) AS remote_lsn
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ remote_lsn_on_subscriber
+--------------------------
+ 0/3000388
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, on the standby server check that the last-received WAL location
+ is ahead of the replayed WAL location(s) on the subscriber identified
+ above. If the above SQL result was NULL, it means the subscriber has not
+ yet replayed any WAL, so the standby server must be ahead of the
+ subscriber, and this step can be skipped.
+<programlisting>
+test_standby=# SELECT pg_last_wal_receive_lsn() >= '0/3000388'::pg_lsn AS failover_ready;
+ failover_ready
+----------------
+ t
+(1 row)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+ </procedure>
+
+ <para>
+ If the result (<literal>failover_ready</literal>) of both above steps is
+ true, existing subscriptions will be able to continue without data loss.
+ </para>
+
+ </sect1>
+
<sect1 id="logical-replication-row-filter">
<title>Row Filters</title>
--
2.30.0.windows.2
[application/octet-stream] v103-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch (45.2K, ../../OS0PR01MB5716B48653125E199A0901D3945D2@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v103-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch)
download | inline diff:
From 99eb2f007039fcfc15b5b0ef3bdced44ec1a6eed Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Fri, 23 Feb 2024 08:29:30 +0800
Subject: [PATCH v103 1/2] Allow logical walsenders to wait for the physical
standby
This patch introduces a mechanism to ensure that physical standby servers,
which are potential failover candidates, have received and flushed changes
before making them visible to subscribers. By doing so, it guarantees that
the promoted standby server is not lagging behind the subscribers when a
failover is necessary.
A new parameter named 'standby_slot_names' is introduced. The logical
walsender now guarantees that all local changes are sent and flushed to
the standby servers corresponding to the replication slots specified in
'standby_slot_names' before sending those changes to the subscriber.
Additionally, The SQL functions pg_logical_slot_get_changes,
pg_logical_slot_peek_changes and pg_replication_slot_advance are modified
to wait for the replication slots mentioned in 'standby_slot_names' to
catch up before returning.
---
doc/src/sgml/config.sgml | 48 +++
doc/src/sgml/func.sgml | 6 +-
doc/src/sgml/logicaldecoding.sgml | 12 +
.../replication/logical/logicalfuncs.c | 12 +
src/backend/replication/logical/slotsync.c | 11 +
src/backend/replication/slot.c | 373 +++++++++++++++++-
src/backend/replication/slotfuncs.c | 12 +
src/backend/replication/walsender.c | 157 +++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 14 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/replication/slot.h | 5 +
src/include/replication/walsender.h | 1 +
src/include/replication/walsender_private.h | 7 +
src/include/utils/guc_hooks.h | 3 +
src/test/recovery/t/006_logical_decoding.pl | 3 +-
.../t/040_standby_failover_slots_sync.pl | 202 ++++++++++
17 files changed, 845 insertions(+), 24 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 43b1a132a2..9396ec4313 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4559,6 +4559,54 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Lists the streaming replication standby server slot names that logical
+ WAL sender processes will wait for. Logical WAL sender processes will
+ send decoded changes to plugins only after the specified replication
+ slots confirm receiving WAL. This guarantees that logical replication
+ slots with failover enabled do not consume changes until those changes
+ are received and flushed to corresponding physical standbys. If a
+ logical replication connection is meant to switch to a physical standby
+ after the standby is promoted, the physical replication slot for the
+ standby should be listed here. Note that logical replication will not
+ proceed if the slots specified in the
+ <varname>standby_slot_names</varname> do not exist or are invalidated.
+ Additionally, when using the replication management functions
+ <link linkend="pg-replication-slot-advance">
+ <function>pg_replication_slot_advance</function></link>,
+ <link linkend="pg-logical-slot-get-changes">
+ <function>pg_logical_slot_get_changes</function></link>, and
+ <link linkend="pg-logical-slot-peek-changes">
+ <function>pg_logical_slot_peek_changes</function></link>,
+ with failover enabled logical slots, the functions will wait for the
+ physical slots specified in <varname>standby_slot_names</varname> to
+ confirm WAL receipt before proceeding.
+ </para>
+ <para>
+ The standbys corresponding to the physical replication slots in
+ <varname>standby_slot_names</varname> must configure
+ <literal>sync_replication_slots = true</literal> so they can receive
+ failover logical slots changes from the primary.
+ </para>
+ <note>
+ <para>
+ Value <literal>*</literal> is not accepted as it is inappropriate to
+ block logical replication for physical slots that either lack
+ associated standbys or have standbys associated that are not enabled
+ for replication slot synchronization. (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>).
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index e5fa82c161..cadea3ceb1 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -28150,7 +28150,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-get-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_get_changes</primary>
</indexterm>
@@ -28177,7 +28177,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-peek-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_peek_changes</primary>
</indexterm>
@@ -28232,7 +28232,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-replication-slot-advance" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_replication_slot_advance</primary>
</indexterm>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 930c0fa8a6..5da3ac7f4c 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -384,6 +384,18 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU
must be enabled on the standby. It is also necessary to specify a valid
<literal>dbname</literal> in the
<link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>.
+ It's highly recommended that the said physical replication slot is named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ list on the primary, to prevent the subscriber from consuming changes
+ faster than the hot standby. Even when correctly configured, some latency
+ is expected when sending changes to logical subscribers due to the waiting
+ on slots named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>.
+ When <varname>standby_slot_names</varname> is utilized, the
+ primary server will not completely shut down until the corresponding
+ standbys, associated with the physical replication slots specified
+ in <varname>standby_slot_names</varname>, have confirmed
+ receiving the WAL up to the latest flushed position on the primary server.
</para>
<para>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b0081d3ce5..f28d6c9015 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -109,6 +109,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
MemoryContext per_query_ctx;
MemoryContext oldcontext;
XLogRecPtr end_of_wal;
+ XLogRecPtr wait_for_wal_lsn;
LogicalDecodingContext *ctx;
ResourceOwner old_resowner = CurrentResourceOwner;
ArrayType *arr;
@@ -228,6 +229,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
NameStr(MyReplicationSlot->data.plugin),
format_procedure(fcinfo->flinfo->fn_oid))));
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to wait_for_wal_lsn.
+ */
+ if (XLogRecPtrIsInvalid(upto_lsn))
+ wait_for_wal_lsn = end_of_wal;
+ else
+ wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
+
+ WaitForStandbyConfirmation(wait_for_wal_lsn);
+
ctx->output_writer_private = p;
/*
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 8ecb85b86a..d1562cf7b3 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -491,6 +491,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
latestFlushPtr = GetStandbyFlushRecPtr(NULL);
if (remote_slot->confirmed_lsn > latestFlushPtr)
{
+ /*
+ * Can get here only if GUC 'standby_slot_names' on the primary server
+ * was not configured correctly.
+ */
ereport(am_slotsync_worker ? LOG : ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("skipping slot synchronization as the received slot sync"
@@ -860,6 +864,13 @@ validate_remote_info(WalReceiverConn *wrconn)
remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull));
Assert(!isnull);
+ /*
+ * Slot sync is currently not supported on a cascading standby. This is
+ * because if we allow it, the primary server needs to wait for all the
+ * cascading standbys, otherwise, logical subscribers can still be ahead
+ * of one of the cascading standbys which we plan to promote. Thus, to
+ * avoid this additional complexity, we restrict it for the time being.
+ */
if (remote_in_recovery)
ereport(ERROR,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 0f173f63a2..98b27e7790 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -46,13 +46,18 @@
#include "common/string.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/interrupt.h"
#include "replication/slotsync.h"
#include "replication/slot.h"
+#include "replication/walsender_private.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
+#include "utils/memutils.h"
+#include "utils/varlena.h"
/*
* Replication slot on-disk data structure.
@@ -115,10 +120,25 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
/* My backend's replication slot in the shared memory array */
ReplicationSlot *MyReplicationSlot = NULL;
-/* GUC variable */
+/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+/*
+ * This GUC lists streaming replication standby server slot names that
+ * logical WAL sender processes will wait for.
+ */
+char *standby_slot_names;
+
+/* This is parsed and cached list for raw standby_slot_names. */
+static List *standby_slot_names_list = NIL;
+
+/*
+ * Oldest LSN that has been confirmed to be flushed to the standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+static XLogRecPtr standby_slot_oldest_flush_lsn = InvalidXLogRecPtr;
+
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
@@ -2345,3 +2365,354 @@ GetSlotInvalidationCause(const char *conflict_reason)
Assert(found);
return result;
}
+
+/*
+ * A helper function to validate slots specified in GUC standby_slot_names.
+ */
+static bool
+validate_standby_slots(char **newval)
+{
+ char *rawname;
+ List *elemlist;
+ bool ok;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Verify syntax and parse string into a list of identifiers */
+ ok = SplitIdentifierString(rawname, ',', &elemlist);
+
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
+ else if (!ReplicationSlotCtl)
+ {
+ /*
+ * We cannot validate the replication slot if the replication slots'
+ * data has not been initialized. This is ok as we will validate the
+ * specified slot when waiting for them to catch up. See
+ * StandbySlotsHaveCaughtup for details.
+ */
+ }
+ else
+ {
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, elemlist)
+ {
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ GUC_check_errdetail("replication slot \"%s\" does not exist",
+ name);
+ ok = false;
+ break;
+ }
+
+ if (!SlotIsPhysical(slot))
+ {
+ GUC_check_errdetail("\"%s\" is not a physical replication slot",
+ name);
+ ok = false;
+ break;
+ }
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+ }
+
+ pfree(rawname);
+ list_free(elemlist);
+ return ok;
+}
+
+/*
+ * GUC check_hook for standby_slot_names
+ */
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ if ((*newval)[0] == '\0')
+ return true;
+
+ /*
+ * "*" is not accepted as in that case primary will not be able to know
+ * for which all standbys to wait for. Even if we have physical slots
+ * info, there is no way to confirm whether there is any standby
+ * configured for the known physical slots.
+ */
+ if (strcmp(*newval, "*") == 0)
+ {
+ GUC_check_errdetail("\"*\" is not accepted for standby_slot_names");
+ return false;
+ }
+
+ /* Now verify if the specified slots really exist and have correct type */
+ if (!validate_standby_slots(newval))
+ return false;
+
+ *extra = guc_strdup(ERROR, *newval);
+
+ return true;
+}
+
+/*
+ * GUC assign_hook for standby_slot_names
+ */
+void
+assign_standby_slot_names(const char *newval, void *extra)
+{
+ List *standby_slots;
+ MemoryContext oldcxt;
+ char *standby_slot_names_cpy = extra;
+
+ /*
+ * The standby slots may have changed, so we need to recompute the oldest
+ * LSN.
+ */
+ standby_slot_oldest_flush_lsn = InvalidXLogRecPtr;
+
+ list_free(standby_slot_names_list);
+ standby_slot_names_list = NIL;
+
+ /* No value is specified for standby_slot_names. */
+ if (standby_slot_names_cpy == NULL)
+ return;
+
+ if (!SplitIdentifierString(standby_slot_names_cpy, ',', &standby_slots))
+ {
+ /* This should not happen if GUC checked check_standby_slot_names. */
+ elog(ERROR, "invalid list syntax");
+ }
+
+ /*
+ * Switch to the memory context under which GUC variables are allocated
+ * (GUCMemoryContext).
+ */
+ oldcxt = MemoryContextSwitchTo(GetMemoryChunkContext(standby_slot_names_cpy));
+ standby_slot_names_list = list_copy(standby_slots);
+ MemoryContextSwitchTo(oldcxt);
+}
+
+/*
+ * Return the standby_slot_names_list.
+ *
+ * Note that since we do not support syncing slots to cascading standbys, we
+ * return NIL if we are running in a standby to indicate that no standby slots
+ * need to be waited for.
+ */
+List *
+GetStandbySlotList(void)
+{
+ if (RecoveryInProgress())
+ return NIL;
+
+ return standby_slot_names_list;
+}
+
+/*
+ * Return true if the slots specified in standby_slot_names have caught up to
+ * the given WAL location, false otherwise.
+ *
+ * The elevel parameter specifies the error level used for logging messages
+ * related to slots that do not exist, are invalidated, or are inactive.
+ */
+bool
+StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel)
+{
+ int caught_up_slot_num = 0;
+ XLogRecPtr min_restart_lsn = InvalidXLogRecPtr;
+
+ /*
+ * Don't need to wait for the standbys to catch up if there is no value in
+ * standby_slot_names.
+ */
+ if (standby_slot_names_list == NIL)
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if we are on a standby
+ * server, since we do not support syncing slots to cascading standbys.
+ */
+ if (RecoveryInProgress())
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if they are already
+ * beyond the specified WAL location.
+ */
+ if (!XLogRecPtrIsInvalid(standby_slot_oldest_flush_lsn) &&
+ standby_slot_oldest_flush_lsn >= wait_for_lsn)
+ return true;
+
+ /*
+ * To prevent concurrent slot dropping and creation while filtering the
+ * slots, take the ReplicationSlotControlLock outside of the loop.
+ */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, standby_slot_names_list)
+ {
+ XLogRecPtr restart_lsn;
+ bool invalidated;
+ bool inactive;
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ /*
+ * If the provided slot does not exist, report a message and exit
+ * the loop. It is possible for a user to specify a slot name in
+ * standby_slot_names that does not exist just before the server
+ * startup. The GUC check_hook(validate_standby_slots) cannot
+ * validate such a slot during startup as the ReplicationSlotCtl
+ * shared memory is not initialized at that time. It is also
+ * possible for a user to drop the slot in standby_slot_names
+ * afterwards.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not exist",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider creating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (SlotIsLogical(slot))
+ {
+ /*
+ * If a logical slot name is provided in standby_slot_names,
+ * report a message and exit the loop. Similar to the non-existent
+ * case, it is possible for a user to specify a logical slot name
+ * in standby_slot_names before the server startup, or drop an
+ * existing physical slot and recreate a logical slot with the
+ * same name.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot have logical replication slot \"%s\" in parameter %s",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting for correction on \"%s\".",
+ name),
+ errhint("Consider removing logical slot \"%s\" from parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ SpinLockAcquire(&slot->mutex);
+ restart_lsn = slot->data.restart_lsn;
+ invalidated = slot->data.invalidated != RS_INVAL_NONE;
+ inactive = slot->active_pid == 0;
+ SpinLockRelease(&slot->mutex);
+
+ if (invalidated)
+ {
+ /* Specified physical slot has been invalidated */
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("physical slot \"%s\" specified in parameter %s has been invalidated",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider dropping and recreating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (XLogRecPtrIsInvalid(restart_lsn) || restart_lsn < wait_for_lsn)
+ {
+ /* Log a message if no active_pid for this physical slot */
+ if (inactive)
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not have active_pid",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider starting standby associated with \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+
+ /* Continue if the current slot hasn't caught up. */
+ break;
+ }
+
+ Assert(restart_lsn >= wait_for_lsn);
+
+ if (XLogRecPtrIsInvalid(min_restart_lsn) ||
+ min_restart_lsn > restart_lsn)
+ min_restart_lsn = restart_lsn;
+
+ caught_up_slot_num++;
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /*
+ * Return false if not all the standbys have caught up to the specified
+ * WAL location.
+ */
+ if (caught_up_slot_num != list_length(standby_slot_names_list))
+ return false;
+
+ Assert(XLogRecPtrIsInvalid(standby_slot_oldest_flush_lsn) ||
+ min_restart_lsn >= standby_slot_oldest_flush_lsn);
+
+ standby_slot_oldest_flush_lsn = min_restart_lsn;
+
+ return true;
+}
+
+/*
+ * Wait for physical standbys to confirm receiving the given lsn.
+ *
+ * Used by logical decoding SQL functions that acquired failover enabled slot.
+ * It waits for physical standbys corresponding to the physical slots specified
+ * in the standby_slot_names GUC.
+ */
+void
+WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
+{
+ /*
+ * Don't need to wait for the standby to catch up if the current acquired
+ * slot is not a failover enabled slot, or there is no value in
+ * standby_slot_names.
+ */
+ if (!MyReplicationSlot->data.failover || standby_slot_names_list == NIL)
+ return;
+
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+
+ for (;;)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /* Exit if done waiting for every slot. */
+ if (StandbySlotsHaveCaughtup(wait_for_lsn, WARNING))
+ break;
+
+ /*
+ * We wait for the slots in the standby_slot_names to catch up, but we
+ * use a timeout (1s) so we can also check if the standby_slot_names
+ * has been changed.
+ */
+ ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000,
+ WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION);
+ }
+
+ ConditionVariableCancelSleep();
+}
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 768a304723..d864fe4133 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -464,6 +464,12 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* crash, but this makes the data consistent after a clean shutdown.
*/
ReplicationSlotMarkDirty();
+
+ /*
+ * Wake up logical walsenders holding failover enabled slots after
+ * updating the restart_lsn of the physical slot.
+ */
+ PhysicalWakeupLogicalWalSnd();
}
return retlsn;
@@ -504,6 +510,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto)
.segment_close = wal_segment_close),
NULL, NULL, NULL);
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to moveto lsn.
+ */
+ WaitForStandbyConfirmation(moveto);
+
/*
* Start reading at the slot's restart_lsn, which we know to point to
* a valid record.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 13bc3e0aee..5345a2c780 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1728,26 +1728,116 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
ProcessPendingWrites();
}
+/*
+ * Wake up the logical walsender processes with failover enabled slots if the
+ * currently acquired physical slot is specified in standby_slot_names GUC.
+ */
+void
+PhysicalWakeupLogicalWalSnd(void)
+{
+ List *standby_slots;
+
+ Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot));
+
+ standby_slots = GetStandbySlotList();
+
+ foreach_ptr(char, name, standby_slots)
+ {
+ if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0)
+ {
+ ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv);
+ return;
+ }
+ }
+}
+
+/*
+ * Returns true if not all standbys have caught up to the flushed position
+ * (flushed_lsn) when failover_slot is true; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForStandby(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
+{
+ int elevel = got_STOPPING ? ERROR : WARNING;
+ bool failover_slot;
+
+ failover_slot = (replication_active && MyReplicationSlot->data.failover);
+
+ if (failover_slot && !StandbySlotsHaveCaughtup(flushed_lsn, elevel))
+ {
+ *wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
+ return true;
+ }
+
+ *wait_event = 0;
+ return false;
+}
+
+/*
+ * Returns true if we need to wait for WALs to be flushed to disk, or if not
+ * all standbys have caught up to the flushed position (flushed_lsn) when
+ * failover_slot is true; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
+{
+ /* Check if we need to wait for WALs to be flushed to disk. */
+ if (target_lsn > flushed_lsn)
+ {
+ *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
+ return true;
+ }
+
+ /*
+ * Check if the standby slots have caught up to the flushed position. It
+ * is good to wait up to flushed position and then let it send the changes
+ * to logical subscribers one by one which are already covered in flushed
+ * position without needing to wait on every change for standby
+ * confirmation. Note that after receiving the shutdown signal, an ERROR
+ * is reported if any slots are dropped, invalidated, or inactive. This
+ * measure is taken to prevent the walsender from waiting indefinitely.
+ */
+ if (NeedToWaitForStandby(target_lsn, flushed_lsn, wait_event))
+ return true;
+
+ *wait_event = 0;
+ return false;
+}
+
/*
* Wait till WAL < loc is flushed to disk so it can be safely sent to client.
*
- * Returns end LSN of flushed WAL. Normally this will be >= loc, but
- * if we detect a shutdown request (either from postmaster or client)
- * we will return early, so caller must always check.
+ * If the walsender holds a logical slot that has enabled failover, we also
+ * wait for all the specified streaming replication standby servers to
+ * confirm receipt of WAL up to RecentFlushPtr.
+ *
+ * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we
+ * detect a shutdown request (either from postmaster or client) we will return
+ * early, so caller must always check.
*/
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
int wakeEvents;
+ uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
- * have enough WAL available. This is particularly interesting if we're
- * far behind.
+ * have enough WAL available and all the standby servers have confirmed
+ * receipt of WAL up to RecentFlushPtr. This is particularly interesting
+ * if we're far behind.
*/
- if (RecentFlushPtr != InvalidXLogRecPtr &&
- loc <= RecentFlushPtr)
+ if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
return RecentFlushPtr;
/* Get a more recent flush pointer. */
@@ -1758,6 +1848,7 @@ WalSndWaitForWal(XLogRecPtr loc)
for (;;)
{
+ bool wait_for_standby_at_stop = false;
long sleeptime;
/* Clear any already-pending wakeups */
@@ -1784,21 +1875,34 @@ WalSndWaitForWal(XLogRecPtr loc)
if (got_STOPPING)
XLogBackgroundFlush();
- /* Update our idea of the currently flushed position. */
- if (!RecoveryInProgress())
- RecentFlushPtr = GetFlushRecPtr(NULL);
- else
- RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * Update our idea of the currently flushed position only if we are
+ * not waiting for standbys to catch up, otherwise the standby would
+ * have to catch up to a newer WAL location in each cycle.
+ */
+ if (wait_event != WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ {
+ if (!RecoveryInProgress())
+ RecentFlushPtr = GetFlushRecPtr(NULL);
+ else
+ RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ }
/*
- * If postmaster asked us to stop, don't wait anymore.
+ * If postmaster asked us to stop and the standby slots have caught up
+ * to the flushed position, don't wait anymore.
*
* It's important to do this check after the recomputation of
* RecentFlushPtr, so we can send all remaining data before shutting
* down.
*/
if (got_STOPPING)
- break;
+ {
+ if (NeedToWaitForStandby(loc, RecentFlushPtr, &wait_event))
+ wait_for_standby_at_stop = true;
+ else
+ break;
+ }
/*
* We only send regular messages to the client for full decoded
@@ -1813,11 +1917,18 @@ WalSndWaitForWal(XLogRecPtr loc)
!waiting_for_ping_response)
WalSndKeepalive(false, InvalidXLogRecPtr);
- /* check whether we're done */
- if (loc <= RecentFlushPtr)
+ /*
+ * Exit the loop if already caught up and doesn't need to wait for
+ * standby slots.
+ */
+ if (!wait_for_standby_at_stop &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
break;
- /* Waiting for new WAL. Since we need to wait, we're now caught up. */
+ /*
+ * Waiting for new WAL or waiting for standbys to catch up. Since we
+ * need to wait, we're now caught up.
+ */
WalSndCaughtUp = true;
/*
@@ -1855,7 +1966,7 @@ WalSndWaitForWal(XLogRecPtr loc)
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL);
+ WalSndWait(wakeEvents, sleeptime, wait_event);
}
/* reactivate latch so WalSndLoop knows to continue */
@@ -2265,6 +2376,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn)
{
ReplicationSlotMarkDirty();
ReplicationSlotsComputeRequiredLSN();
+ PhysicalWakeupLogicalWalSnd();
}
/*
@@ -3538,6 +3650,7 @@ WalSndShmemInit(void)
ConditionVariableInit(&WalSndCtl->wal_flush_cv);
ConditionVariableInit(&WalSndCtl->wal_replay_cv);
+ ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv);
}
}
@@ -3607,8 +3720,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event)
*
* And, we use separate shared memory CVs for physical and logical
* walsenders for selective wake ups, see WalSndWakeup() for more details.
+ *
+ * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV
+ * until awakened by physical walsenders after the walreceiver confirms
+ * the receipt of the LSN.
*/
- if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
+ if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+ else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv);
else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index ec2f31f82a..c08e00d1d6 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP
LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server."
LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server."
SSL_OPEN_SERVER "Waiting for SSL while attempting connection."
+WAIT_FOR_STANDBY_CONFIRMATION "Waiting for WAL to be received and flushed by the physical standby."
WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process."
WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 93ded31ed9..6741b2a10c 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4670,6 +4670,20 @@ struct config_string ConfigureNamesString[] =
check_debug_io_direct, assign_debug_io_direct, NULL
},
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("Lists streaming replication standby server slot "
+ "names that logical WAL sender processes will wait for."),
+ gettext_noop("Logical WAL sender processes will send decoded "
+ "changes to plugins only after the specified "
+ "replication slots confirm receiving WAL."),
+ GUC_LIST_INPUT
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, assign_standby_slot_names, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index edcc0282b2..2244ee52f7 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -343,6 +343,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsender processes will wait for
# - Standby Servers -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index acbf567150..61a6b97952 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -226,6 +226,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
+extern PGDLLIMPORT char *standby_slot_names;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -274,4 +275,8 @@ extern void CheckSlotPermissions(void);
extern ReplicationSlotInvalidationCause
GetSlotInvalidationCause(const char *conflict_reason);
+extern List *GetStandbySlotList(void);
+extern bool StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel);
+extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn);
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index 0c3996e926..f2d8297f01 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -39,6 +39,7 @@ extern void InitWalSender(void);
extern bool exec_replication_command(const char *cmd_string);
extern void WalSndErrorCleanup(void);
extern void WalSndResourceCleanup(bool isCommit);
+extern void PhysicalWakeupLogicalWalSnd(void);
extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
extern void WalSndSignals(void);
extern Size WalSndShmemSize(void);
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 3113e9ea47..3c134c8edf 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -113,6 +113,13 @@ typedef struct
ConditionVariable wal_flush_cv;
ConditionVariable wal_replay_cv;
+ /*
+ * Used by physical walsenders holding slots specified in
+ * standby_slot_names to wake up logical walsenders holding failover
+ * enabled slots when a walreceiver confirms the receipt of LSN.
+ */
+ ConditionVariable wal_confirm_rcv_cv;
+
WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER];
} WalSndCtlData;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index c8a7aa9a11..d64dc5fcdb 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra,
extern void assign_wal_consistency_checking(const char *newval, void *extra);
extern bool check_wal_segment_size(int *newval, void **extra, GucSource source);
extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
+extern bool check_standby_slot_names(char **newval, void **extra,
+ GucSource source);
+extern void assign_standby_slot_names(const char *newval, void *extra);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl
index 5c7b4ca5e3..b95d95c06f 100644
--- a/src/test/recovery/t/006_logical_decoding.pl
+++ b/src/test/recovery/t/006_logical_decoding.pl
@@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'},
undef, 'logical slot was actually dropped with DB');
# Test logical slot advancing and its durability.
+# Passing failover=true (last arg) should not have any impact on advancing.
my $logical_slot = 'logical_slot';
$node_primary->safe_psql('postgres',
- "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);"
+ "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);"
);
$node_primary->psql(
'postgres', "
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 021c58f621..5e4dd6c0ab 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -504,11 +504,213 @@ ok( $standby1->poll_query_until(
"SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot' AND synced AND NOT temporary;"),
'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby');
+##################################################
+# Test primary disallowing specified logical replication slots getting ahead of
+# specified physical replication slots. It uses the following set up:
+#
+# (physical standbys)
+# | ----> standby1 (primary_slot_name = sb1_slot)
+# | ----> standby2 (primary_slot_name = sb2_slot)
+# primary ----- |
+# (logical replication)
+# | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
+# | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
+#
+# standby_slot_names = 'sb1_slot'
+#
+# Set up is configured in such a way that the logical slot of subscriber1 is
+# enabled for failover, thus it will wait for the physical slot of
+# standby1(sb1_slot) to catch up before sending decoded changes to subscriber1.
+##################################################
+
+$backup_name = 'backup3';
+
+$primary->psql('postgres',
+ q{SELECT pg_create_physical_replication_slot('sb2_slot');});
+
+$primary->backup($backup_name);
+
+# Create another standby
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup(
+ $primary, $backup_name,
+ has_streaming => 1,
+ has_restoring => 1);
+$standby2->append_conf(
+ 'postgresql.conf', qq(
+primary_slot_name = 'sb2_slot'
+));
+$standby2->start;
+$primary->wait_for_replay_catchup($standby2);
+
+# Configure primary to disallow any logical slots that enabled failover from
+# getting ahead of specified physical replication slot (sb1_slot).
+$primary->append_conf(
+ 'postgresql.conf', qq(
+standby_slot_names = 'sb1_slot'
+));
+$primary->reload;
+
+# Create another subscriber node without enabling failover, wait for sync to
+# complete
+my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$subscriber2->init;
+$subscriber2->start;
+$subscriber2->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE tab_int (a int PRIMARY KEY);
+ CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot);
+]);
+
+$subscriber2->wait_for_subscription_sync;
+
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+# Stop the standby associated with the specified physical replication slot
+# (sb1_slot) so that the logical replication slot (lsub1_slot) won't receive
+# changes until the standby comes up.
+$standby1->stop;
+
+# Create some data on the primary
+my $primary_row_count = 20;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(11, $primary_row_count);");
+
+# Wait until the standby2 that's still running gets the data from the primary
+$primary->wait_for_replay_catchup($standby2);
+$result = $standby2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby2 gets data from primary");
+
+# Wait for regress_mysub2 to get the data from the primary. This subscription
+# was not enabled for failover so it gets the data without waiting for any
+# standbys.
+$primary->wait_for_catchup('regress_mysub2');
+$result = $subscriber2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "subscriber2 gets data from primary");
+
+# The regress_mysub1 was enabled for failover so it doesn't get the data from
+# primary and keeps waiting for the standby specified in standby_slot_names
+# (sb1_slot aka standby1).
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) <> $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data from primary until standby1 acknowledges changes"
+);
+
+# Start the standby specified in standby_slot_names (sb1_slot aka standby1) and
+# wait for it to catch up with the primary.
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+$result = $standby1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby1 gets data from primary");
+
+# Now that the standby specified in standby_slot_names is up and running, the
+# primary can send the decoded changes to the subscription enabled for failover
+# (i.e. regress_mysub1). While the standby was down, regress_mysub1 didn't
+# receive any data from the primary. i.e. the primary didn't allow it to go
+# ahead of standby.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 acknowledges changes");
+
+##################################################
+# Verify that when using pg_logical_slot_get_changes to consume changes from a
+# logical slot with failover enabled, it will also wait for the slots specified
+# in standby_slot_names to catch up.
+##################################################
+
+# Stop the standby associated with the specified physical replication slot so
+# that the logical replication slot won't receive changes until the standby
+# slot's restart_lsn is advanced or the slot is removed from the
+# standby_slot_names list.
+$primary->safe_psql('postgres', "TRUNCATE tab_int;");
+$primary->wait_for_catchup('regress_mysub1');
+$standby1->stop;
+
+# Create a logical 'test_decoding' replication slot with failover enabled
+$primary->safe_psql('postgres',
+ "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);"
+);
+
+my $back_q = $primary->background_psql(
+ 'postgres',
+ on_error_stop => 0,
+ timeout => $PostgreSQL::Test::Utils::timeout_default);
+
+# Try and get changes from the logical slot with failover enabled.
+my $offset = -s $primary->logfile;
+$back_q->query_until(qr//,
+ "SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);\n");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the function
+# pg_logical_slot_get_changes should now return, and the session can be
+# stopped.
+$back_q->quit;
+
+$primary->safe_psql('postgres',
+ "SELECT pg_drop_replication_slot('test_slot');"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
+##################################################
+# Test that logical replication will wait for the user-created inactive
+# physical slot to catch up until we remove the slot from standby_slot_names.
+##################################################
+
+# Create some data on the primary
+$primary_row_count = 10;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);");
+
+# The regress_mysub1 doesn't get the data from primary because the specified
+# standby slot (sb1_slot) in standby_slot_names is inactive.
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data as the sb1_slot doesn't catch up");
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the primary server should now
+# send the decoded changes to the subscription.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list"
+);
+
##################################################
# Promote the standby1 to primary. Confirm that:
# a) the slot 'lsub1_slot' is retained on the new primary
# b) logical replication for regress_mysub1 is resumed successfully after failover
##################################################
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+
$standby1->promote;
# Update subscription with the new primary's connection info
--
2.30.0.windows.2
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-02 03:51 Zhijie Hou (Fujitsu) <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 0 replies; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-02 03:51 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Thursday, February 29, 2024 11:16 AM Zhijie Hou (Fujitsu) <[email protected]> wrote:
>
> On Wednesday, February 28, 2024 7:36 PM Bertrand Drouvot
> <[email protected]> wrote:
> >
> > 4 ===
> >
> > Regarding the test, what about adding one to test the "new" behavior
> > discussed up-thread? (logical replication will wait if slot mentioned
> > in standby_slot_names is dropped and/or does not exist when the engine
> > starts?)
>
> Will think about this.
I think we could add a test to check the warning message for dropped slot, but
since similar wait/warning functionality has been tested, I prefer to leave
this for now and can consider it again after the main patch and tests are
stabilized considering the previous experience of BF instability with this
feature.
Best Regards,
Hou zj
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-02 10:54 Amit Kapila <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
1 sibling, 1 reply; 68+ messages in thread
From: Amit Kapila @ 2024-03-02 10:54 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Peter Smith <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Sat, Mar 2, 2024 at 9:21 AM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> Apart from the comments, the code in WalSndWaitForWal was refactored
> a bit to make it neater. Thanks Shveta for helping writing the code and doc.
>
A few more comments:
==================
1.
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter
standby_slot_names does not have active_pid/,
+ $offset);
Shouldn't we wait for such a LOG even in the first test as well which
involves two standbys and two logical subscribers?
2.
+##################################################
+# Test that logical replication will wait for the user-created inactive
+# physical slot to catch up until we remove the slot from standby_slot_names.
+##################################################
I don't see anything different tested in this test from what we
already tested in the first test involving two standbys and two
logical subscribers. Can you please clarify if I am missing something?
3.
Note that after receiving the shutdown signal, an ERROR
+ * is reported if any slots are dropped, invalidated, or inactive. This
+ * measure is taken to prevent the walsender from waiting indefinitely.
+ */
+ if (NeedToWaitForStandby(target_lsn, flushed_lsn, wait_event))
Isn't this part of the comment should be moved inside NeedToWaitForStandby()?
4.
+ /*
+ * Update our idea of the currently flushed position only if we are
+ * not waiting for standbys to catch up, otherwise the standby would
+ * have to catch up to a newer WAL location in each cycle.
+ */
+ if (wait_event != WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ {
This functionality (in function WalSndWaitForWal()) seems to ensure
that we first wait for the required WAL to be flushed and then wait
for standbys. If true, we should cover that point in the comments here
or somewhere in the function WalSndWaitForWal().
Apart from this, I have made a few modifications in the comments.
--
With Regards,
Amit Kapila.
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 5e4dd6c0ab..16483e52a9 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -505,8 +505,9 @@ ok( $standby1->poll_query_until(
'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby');
##################################################
-# Test primary disallowing specified logical replication slots getting ahead of
-# specified physical replication slots. It uses the following set up:
+# Test that logical failover replication slots wait for the specified
+# physical replication slots to receive the changes first. It uses the
+# following set up:
#
# (physical standbys)
# | ----> standby1 (primary_slot_name = sb1_slot)
@@ -518,9 +519,9 @@ ok( $standby1->poll_query_until(
#
# standby_slot_names = 'sb1_slot'
#
-# Set up is configured in such a way that the logical slot of subscriber1 is
-# enabled for failover, thus it will wait for the physical slot of
-# standby1(sb1_slot) to catch up before sending decoded changes to subscriber1.
+# The setup is configured in such a way that the logical slot of subscriber1 is
+# enabled for failover, and thus the subscriber1 will wait for the physical
+# slot of standby1(sb1_slot) to catch up before receiving the decoded changes.
##################################################
$backup_name = 'backup3';
@@ -543,8 +544,8 @@ primary_slot_name = 'sb2_slot'
$standby2->start;
$primary->wait_for_replay_catchup($standby2);
-# Configure primary to disallow any logical slots that enabled failover from
-# getting ahead of specified physical replication slot (sb1_slot).
+# Configure primary to disallow any logical slots that have enabled failover
+# from getting ahead of the specified physical replication slot (sb1_slot).
$primary->append_conf(
'postgresql.conf', qq(
standby_slot_names = 'sb1_slot'
Attachments:
[text/plain] v103-0001_amit_1.patch.txt (2.0K, ../../CAA4eK1KpFuUPkaFDMXoUOujGVHj4yGWxQ_Xx3DSUU09ieR12Fg@mail.gmail.com/2-v103-0001_amit_1.patch.txt)
download | inline diff:
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 5e4dd6c0ab..16483e52a9 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -505,8 +505,9 @@ ok( $standby1->poll_query_until(
'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby');
##################################################
-# Test primary disallowing specified logical replication slots getting ahead of
-# specified physical replication slots. It uses the following set up:
+# Test that logical failover replication slots wait for the specified
+# physical replication slots to receive the changes first. It uses the
+# following set up:
#
# (physical standbys)
# | ----> standby1 (primary_slot_name = sb1_slot)
@@ -518,9 +519,9 @@ ok( $standby1->poll_query_until(
#
# standby_slot_names = 'sb1_slot'
#
-# Set up is configured in such a way that the logical slot of subscriber1 is
-# enabled for failover, thus it will wait for the physical slot of
-# standby1(sb1_slot) to catch up before sending decoded changes to subscriber1.
+# The setup is configured in such a way that the logical slot of subscriber1 is
+# enabled for failover, and thus the subscriber1 will wait for the physical
+# slot of standby1(sb1_slot) to catch up before receiving the decoded changes.
##################################################
$backup_name = 'backup3';
@@ -543,8 +544,8 @@ primary_slot_name = 'sb2_slot'
$standby2->start;
$primary->wait_for_replay_catchup($standby2);
-# Configure primary to disallow any logical slots that enabled failover from
-# getting ahead of specified physical replication slot (sb1_slot).
+# Configure primary to disallow any logical slots that have enabled failover
+# from getting ahead of the specified physical replication slot (sb1_slot).
$primary->append_conf(
'postgresql.conf', qq(
standby_slot_names = 'sb1_slot'
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-02 23:46 Peter Smith <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
1 sibling, 2 replies; 68+ messages in thread
From: Peter Smith @ 2024-03-02 23:46 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Sat, Mar 2, 2024 at 2:51 PM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> On Friday, March 1, 2024 12:23 PM Peter Smith <[email protected]> wrote:
> >
...
> > ======
> > src/backend/replication/slot.c
> >
> > 2. validate_standby_slots
> >
> > + else if (!ReplicationSlotCtl)
> > + {
> > + /*
> > + * We cannot validate the replication slot if the replication slots'
> > + * data has not been initialized. This is ok as we will validate the
> > + * specified slot when waiting for them to catch up. See
> > + * StandbySlotsHaveCaughtup for details.
> > + */
> > + }
> > + else
> > + {
> > + /*
> > + * If the replication slots' data have been initialized, verify if the
> > + * specified slots exist and are logical slots.
> > + */
> > + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
> >
> > IMO that 2nd comment does not need to say "If the replication slots'
> > data have been initialized," because that is implicit from the if/else.
>
> Removed.
I only meant to suggest removing the 1st part, not the entire comment.
I thought it is still useful to have a comment like:
/* Check that the specified slots exist and are logical slots.*/
======
And, here are some review comments for v103-0001.
======
Commit message
1.
Additionally, The SQL functions pg_logical_slot_get_changes,
pg_logical_slot_peek_changes and pg_replication_slot_advance are modified
to wait for the replication slots mentioned in 'standby_slot_names' to
catch up before returning.
~
(use the same wording as previous in this message)
/mentioned in/specified in/
======
doc/src/sgml/config.sgml
2.
+ Additionally, when using the replication management functions
+ <link linkend="pg-replication-slot-advance">
+ <function>pg_replication_slot_advance</function></link>,
+ <link linkend="pg-logical-slot-get-changes">
+ <function>pg_logical_slot_get_changes</function></link>, and
+ <link linkend="pg-logical-slot-peek-changes">
+ <function>pg_logical_slot_peek_changes</function></link>,
+ with failover enabled logical slots, the functions will wait for the
+ physical slots specified in <varname>standby_slot_names</varname> to
+ confirm WAL receipt before proceeding.
+ </para>
It says "the ... functions" twice so maybe reword it slightly.
BEFORE
Additionally, when using the replication management functions
pg_replication_slot_advance, pg_logical_slot_get_changes, and
pg_logical_slot_peek_changes, with failover enabled logical slots, the
functions will wait for the physical slots specified in
standby_slot_names to confirm WAL receipt before proceeding.
SUGGESTION
Additionally, the replication management functions
pg_replication_slot_advance, pg_logical_slot_get_changes, and
pg_logical_slot_peek_changes, when used with failover enabled logical
slots, will wait for the physical slots specified in
standby_slot_names to confirm WAL receipt before proceeding.
(Actually the "will wait ... before proceeding" is also a bit tricky,
so below is another possible rewording)
SUGGESTION #2
Additionally, the replication management functions
pg_replication_slot_advance, pg_logical_slot_get_changes, and
pg_logical_slot_peek_changes, when used with failover enabled logical
slots, will block until all physical slots specified in
standby_slot_names have confirmed WAL receipt.
~~~
3.
+ <note>
+ <para>
+ Value <literal>*</literal> is not accepted as it is inappropriate to
+ block logical replication for physical slots that either lack
+ associated standbys or have standbys associated that are not enabled
+ for replication slot synchronization. (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>).
+ </para>
+ </note>
Why does the document need to provide an excuse/reason for the rule?
You could just say something like:
SUGGESTION
The slots must be named explicitly. For example, specifying wildcard
values like <literal>*</literal> is not permitted.
======
doc/src/sgml/func.sgml
4.
@@ -28150,7 +28150,7 @@ postgres=# SELECT '0/0'::pg_lsn +
pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-get-changes"
role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_get_changes</primary>
</indexterm>
@@ -28177,7 +28177,7 @@ postgres=# SELECT '0/0'::pg_lsn +
pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-peek-changes"
role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_peek_changes</primary>
</indexterm>
@@ -28232,7 +28232,7 @@ postgres=# SELECT '0/0'::pg_lsn +
pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-replication-slot-advance"
role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_replication_slot_advance</primary>
</indexterm>
Should these 3 functions say something about how their behaviour is
affected by 'standby_slot_names' and give a link back to the GUC
'standby_slot_names' docs?
======
src/backend/replication/slot.c
5. StandbySlotsHaveCaughtup
+ if (!slot)
+ {
+ /*
+ * If the provided slot does not exist, report a message and exit
+ * the loop. It is possible for a user to specify a slot name in
+ * standby_slot_names that does not exist just before the server
+ * startup. The GUC check_hook(validate_standby_slots) cannot
+ * validate such a slot during startup as the ReplicationSlotCtl
+ * shared memory is not initialized at that time. It is also
+ * possible for a user to drop the slot in standby_slot_names
+ * afterwards.
+ */
5a.
Minor rewording to make this code comment more similar to the next one:
SUGGESTION
If a slot name provided in standby_slot_names does not exist, report a
message and exit the loop. A user can specify a slot name that does
not exist just before the server startup...
~
5b.
+ /*
+ * If a logical slot name is provided in standby_slot_names,
+ * report a message and exit the loop. Similar to the non-existent
+ * case, it is possible for a user to specify a logical slot name
+ * in standby_slot_names before the server startup, or drop an
+ * existing physical slot and recreate a logical slot with the
+ * same name.
+ */
/it is possible for a user to specify/a user can specify/
~~~
6. WaitForStandbyConfirmation
+ /*
+ * We wait for the slots in the standby_slot_names to catch up, but we
+ * use a timeout (1s) so we can also check if the standby_slot_names
+ * has been changed.
+ */
Remove some of the "we".
SUGGESTION
Wait for the slots in the standby_slot_names to catch up, but use a
timeout (1s) so we can also check if the standby_slot_names has been
changed.
======
src/backend/replication/walsender.c
7. NeedToWaitForStandby
+/*
+ * Returns true if not all standbys have caught up to the flushed position
+ * (flushed_lsn) when failover_slot is true; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
The function comment refers to 'failover_slot' but IMO needs to be
worded differently because 'failover_slot' is not a parameter anymore.
~~~
8. NeedToWaitForWal
+/*
+ * Returns true if we need to wait for WALs to be flushed to disk, or if not
+ * all standbys have caught up to the flushed position (flushed_lsn) when
+ * failover_slot is true; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
Same as above -- Now that 'failover_slot' is not a function parameter,
I thought this should be reworded.
~~~
9. NeedToWaitForWal
+ /*
+ * Check if the standby slots have caught up to the flushed position. It
+ * is good to wait up to flushed position and then let it send the changes
+ * to logical subscribers one by one which are already covered in flushed
+ * position without needing to wait on every change for standby
+ * confirmation. Note that after receiving the shutdown signal, an ERROR
+ * is reported if any slots are dropped, invalidated, or inactive. This
+ * measure is taken to prevent the walsender from waiting indefinitely.
+ */
+ if (NeedToWaitForStandby(target_lsn, flushed_lsn, wait_event))
+ return true;
I felt it was confusing things for this function to also call to the
other one -- it seems an overlapping/muddling of the purpose of these.
I think it will be easier to understand if the calling code just calls
to one or both of these functions as required.
~~~
10.
- WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL);
+ WalSndWait(wakeEvents, sleeptime, wait_event);
Tracing the assignments of the 'wait_event' is a bit tricky... IIUC it
should not be 0 when we got to this point, so maybe it is better to
put Assert(wait_event) before this call?
----------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-03 03:56 Amit Kapila <[email protected]>
parent: Peter Smith <[email protected]>
1 sibling, 1 reply; 68+ messages in thread
From: Amit Kapila @ 2024-03-03 03:56 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Sun, Mar 3, 2024 at 5:17 AM Peter Smith <[email protected]> wrote:
>
> ~~~
>
> 3.
> + <note>
> + <para>
> + Value <literal>*</literal> is not accepted as it is inappropriate to
> + block logical replication for physical slots that either lack
> + associated standbys or have standbys associated that are not enabled
> + for replication slot synchronization. (see
> + <xref linkend="logicaldecoding-replication-slots-synchronization"/>).
> + </para>
> + </note>
>
> Why does the document need to provide an excuse/reason for the rule?
> You could just say something like:
>
> SUGGESTION
> The slots must be named explicitly. For example, specifying wildcard
> values like <literal>*</literal> is not permitted.
>
I would like to document the reason somewhere, if not in docs, then
let's write a comment for the same in code.
> ======
>
> ~~~
>
> 9. NeedToWaitForWal
>
> + /*
> + * Check if the standby slots have caught up to the flushed position. It
> + * is good to wait up to flushed position and then let it send the changes
> + * to logical subscribers one by one which are already covered in flushed
> + * position without needing to wait on every change for standby
> + * confirmation. Note that after receiving the shutdown signal, an ERROR
> + * is reported if any slots are dropped, invalidated, or inactive. This
> + * measure is taken to prevent the walsender from waiting indefinitely.
> + */
> + if (NeedToWaitForStandby(target_lsn, flushed_lsn, wait_event))
> + return true;
>
> I felt it was confusing things for this function to also call to the
> other one -- it seems an overlapping/muddling of the purpose of these.
> I think it will be easier to understand if the calling code just calls
> to one or both of these functions as required.
>
I felt otherwise because the caller has to call these functions at
more than one place which makes the caller's code difficult to follow.
It is better to encapsulate the computation of wait_event.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-03 07:51 Zhijie Hou (Fujitsu) <[email protected]>
parent: Peter Smith <[email protected]>
1 sibling, 1 reply; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-03 07:51 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Sunday, March 3, 2024 7:47 AM Peter Smith <[email protected]> wrote:
>
> On Sat, Mar 2, 2024 at 2:51 PM Zhijie Hou (Fujitsu) <[email protected]>
> wrote:
> >
> > On Friday, March 1, 2024 12:23 PM Peter Smith <[email protected]>
> wrote:
> > >
> ...
> > > ======
> > > src/backend/replication/slot.c
> > >
> > > 2. validate_standby_slots
> > >
> > > + else if (!ReplicationSlotCtl)
> > > + {
> > > + /*
> > > + * We cannot validate the replication slot if the replication slots'
> > > + * data has not been initialized. This is ok as we will validate
> > > + the
> > > + * specified slot when waiting for them to catch up. See
> > > + * StandbySlotsHaveCaughtup for details.
> > > + */
> > > + }
> > > + else
> > > + {
> > > + /*
> > > + * If the replication slots' data have been initialized, verify if
> > > + the
> > > + * specified slots exist and are logical slots.
> > > + */
> > > + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
> > >
> > > IMO that 2nd comment does not need to say "If the replication slots'
> > > data have been initialized," because that is implicit from the if/else.
> >
> > Removed.
>
> I only meant to suggest removing the 1st part, not the entire comment.
> I thought it is still useful to have a comment like:
>
> /* Check that the specified slots exist and are logical slots.*/
OK, I misunderstood. Fixed.
>
> ======
>
> And, here are some review comments for v103-0001.
Thanks for the comments.
>
> ======
> Commit message
>
> 1.
> Additionally, The SQL functions pg_logical_slot_get_changes,
> pg_logical_slot_peek_changes and pg_replication_slot_advance are modified
> to wait for the replication slots mentioned in 'standby_slot_names' to catch up
> before returning.
>
> ~
>
> (use the same wording as previous in this message)
>
> /mentioned in/specified in/
Changed.
>
> ======
> doc/src/sgml/config.sgml
>
> 2.
> + Additionally, when using the replication management functions
> + <link linkend="pg-replication-slot-advance">
> + <function>pg_replication_slot_advance</function></link>,
> + <link linkend="pg-logical-slot-get-changes">
> + <function>pg_logical_slot_get_changes</function></link>, and
> + <link linkend="pg-logical-slot-peek-changes">
> + <function>pg_logical_slot_peek_changes</function></link>,
> + with failover enabled logical slots, the functions will wait for the
> + physical slots specified in
> <varname>standby_slot_names</varname> to
> + confirm WAL receipt before proceeding.
> + </para>
>
> It says "the ... functions" twice so maybe reword it slightly.
>
> BEFORE
> Additionally, when using the replication management functions
> pg_replication_slot_advance, pg_logical_slot_get_changes, and
> pg_logical_slot_peek_changes, with failover enabled logical slots, the functions
> will wait for the physical slots specified in standby_slot_names to confirm WAL
> receipt before proceeding.
>
> SUGGESTION
> Additionally, the replication management functions
> pg_replication_slot_advance, pg_logical_slot_get_changes, and
> pg_logical_slot_peek_changes, when used with failover enabled logical slots,
> will wait for the physical slots specified in standby_slot_names to confirm WAL
> receipt before proceeding.
>
> (Actually the "will wait ... before proceeding" is also a bit tricky, so below is
> another possible rewording)
>
> SUGGESTION #2
> Additionally, the replication management functions
> pg_replication_slot_advance, pg_logical_slot_get_changes, and
> pg_logical_slot_peek_changes, when used with failover enabled logical slots,
> will block until all physical slots specified in standby_slot_names have
> confirmed WAL receipt.
I prefer the #2 version.
>
> ~~~
>
> 3.
> + <note>
> + <para>
> + Value <literal>*</literal> is not accepted as it is inappropriate to
> + block logical replication for physical slots that either lack
> + associated standbys or have standbys associated that are not
> enabled
> + for replication slot synchronization. (see
> + <xref
> linkend="logicaldecoding-replication-slots-synchronization"/>).
> + </para>
> + </note>
>
> Why does the document need to provide an excuse/reason for the rule?
> You could just say something like:
>
> SUGGESTION
> The slots must be named explicitly. For example, specifying wildcard values like
> <literal>*</literal> is not permitted.
As suggested by Amit, I moved this to code comments.
>
> ======
> doc/src/sgml/func.sgml
>
> 4.
> @@ -28150,7 +28150,7 @@ postgres=# SELECT '0/0'::pg_lsn +
> pd.segment_number * ps.setting::int + :offset
> </row>
>
> <row>
> - <entry role="func_table_entry"><para role="func_signature">
> + <entry id="pg-logical-slot-get-changes"
> role="func_table_entry"><para role="func_signature">
> <indexterm>
> <primary>pg_logical_slot_get_changes</primary>
> </indexterm>
> @@ -28177,7 +28177,7 @@ postgres=# SELECT '0/0'::pg_lsn +
> pd.segment_number * ps.setting::int + :offset
> </row>
>
> <row>
> - <entry role="func_table_entry"><para role="func_signature">
> + <entry id="pg-logical-slot-peek-changes"
> role="func_table_entry"><para role="func_signature">
> <indexterm>
> <primary>pg_logical_slot_peek_changes</primary>
> </indexterm>
> @@ -28232,7 +28232,7 @@ postgres=# SELECT '0/0'::pg_lsn +
> pd.segment_number * ps.setting::int + :offset
> </row>
>
> <row>
> - <entry role="func_table_entry"><para role="func_signature">
> + <entry id="pg-replication-slot-advance"
> role="func_table_entry"><para role="func_signature">
> <indexterm>
> <primary>pg_replication_slot_advance</primary>
> </indexterm>
>
> Should these 3 functions say something about how their behaviour is affected
> by 'standby_slot_names' and give a link back to the GUC 'standby_slot_names'
> docs?
I added the info for pg_logical_slot_get_changes() and
pg_replication_slot_advance(). The pg_logical_slot_peek_changes() function
clarifies that it behaves just like pg_logical_slot_get_changes(), so I didn’t
touch it.
>
> ======
> src/backend/replication/slot.c
>
> 5. StandbySlotsHaveCaughtup
>
> + if (!slot)
> + {
> + /*
> + * If the provided slot does not exist, report a message and exit
> + * the loop. It is possible for a user to specify a slot name in
> + * standby_slot_names that does not exist just before the server
> + * startup. The GUC check_hook(validate_standby_slots) cannot
> + * validate such a slot during startup as the ReplicationSlotCtl
> + * shared memory is not initialized at that time. It is also
> + * possible for a user to drop the slot in standby_slot_names
> + * afterwards.
> + */
>
> 5a.
> Minor rewording to make this code comment more similar to the next one:
>
> SUGGESTION
> If a slot name provided in standby_slot_names does not exist, report a message
> and exit the loop. A user can specify a slot name that does not exist just before
> the server startup...
Changed.
>
> ~
>
> 5b.
> + /*
> + * If a logical slot name is provided in standby_slot_names,
> + * report a message and exit the loop. Similar to the non-existent
> + * case, it is possible for a user to specify a logical slot name
> + * in standby_slot_names before the server startup, or drop an
> + * existing physical slot and recreate a logical slot with the
> + * same name.
> + */
>
> /it is possible for a user to specify/a user can specify/
Changed.
>
> ~~~
>
> 6. WaitForStandbyConfirmation
>
> + /*
> + * We wait for the slots in the standby_slot_names to catch up, but we
> + * use a timeout (1s) so we can also check if the standby_slot_names
> + * has been changed.
> + */
>
> Remove some of the "we".
>
> SUGGESTION
> Wait for the slots in the standby_slot_names to catch up, but use a timeout (1s)
> so we can also check if the standby_slot_names has been changed.
Changed.
>
> ======
> src/backend/replication/walsender.c
>
> 7. NeedToWaitForStandby
>
> +/*
> + * Returns true if not all standbys have caught up to the flushed
> +position
> + * (flushed_lsn) when failover_slot is true; otherwise, returns false.
> + *
> + * If returning true, the function sets the appropriate wait event in
> + * wait_event; otherwise, wait_event is set to 0.
> + */
>
> The function comment refers to 'failover_slot' but IMO needs to be worded
> differently because 'failover_slot' is not a parameter anymore.
Changed.
>
> ~~~
>
> 8. NeedToWaitForWal
>
> +/*
> + * Returns true if we need to wait for WALs to be flushed to disk, or
> +if not
> + * all standbys have caught up to the flushed position (flushed_lsn)
> +when
> + * failover_slot is true; otherwise, returns false.
> + *
> + * If returning true, the function sets the appropriate wait event in
> + * wait_event; otherwise, wait_event is set to 0.
> + */
> +static bool
> +NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
> + uint32 *wait_event)
>
> Same as above -- Now that 'failover_slot' is not a function parameter, I thought
> this should be reworded.
Changed.
>
> ~~~
>
> 9. NeedToWaitForWal
>
> + /*
> + * Check if the standby slots have caught up to the flushed position.
> + It
> + * is good to wait up to flushed position and then let it send the
> + changes
> + * to logical subscribers one by one which are already covered in
> + flushed
> + * position without needing to wait on every change for standby
> + * confirmation. Note that after receiving the shutdown signal, an
> + ERROR
> + * is reported if any slots are dropped, invalidated, or inactive. This
> + * measure is taken to prevent the walsender from waiting indefinitely.
> + */
> + if (NeedToWaitForStandby(target_lsn, flushed_lsn, wait_event)) return
> + true;
>
> I felt it was confusing things for this function to also call to the other one -- it
> seems an overlapping/muddling of the purpose of these.
> I think it will be easier to understand if the calling code just calls to one or both
> of these functions as required.
Same as Amit, I didn't change this.
>
> ~~~
>
> 10.
>
> - WalSndWait(wakeEvents, sleeptime,
> WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL);
> + WalSndWait(wakeEvents, sleeptime, wait_event);
>
> Tracing the assignments of the 'wait_event' is a bit tricky... IIUC it should not be
> 0 when we got to this point, so maybe it is better to put Assert(wait_event)
> before this call?
Added.
Best Regards,
Hou zj
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-03 07:56 Zhijie Hou (Fujitsu) <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-03 07:56 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Peter Smith <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Saturday, March 2, 2024 6:55 PM Amit Kapila <[email protected]> wrote:
>
> On Sat, Mar 2, 2024 at 9:21 AM Zhijie Hou (Fujitsu) <[email protected]>
> wrote:
> >
> > Apart from the comments, the code in WalSndWaitForWal was refactored a
> > bit to make it neater. Thanks Shveta for helping writing the code and doc.
> >
>
> A few more comments:
Thanks for the comments.
> ==================
> 1.
> +# Wait until the primary server logs a warning indicating that it is
> +waiting # for the sb1_slot to catch up.
> +$primary->wait_for_log(
> + qr/replication slot \"sb1_slot\" specified in parameter
> standby_slot_names does not have active_pid/,
> + $offset);
>
> Shouldn't we wait for such a LOG even in the first test as well which involves two
> standbys and two logical subscribers?
Yes, we should. Added.
>
> 2.
> +##################################################
> +# Test that logical replication will wait for the user-created inactive
> +# physical slot to catch up until we remove the slot from standby_slot_names.
> +##################################################
>
>
> I don't see anything different tested in this test from what we already tested in
> the first test involving two standbys and two logical subscribers. Can you
> please clarify if I am missing something?
I think the intention was to test that the wait loop is ended due to GUC config
reload, while the first test is for the case when the loop is ended due to
restart_lsn movement. But it seems we tested the config reload with xx_get_changes() as
well, so I can remove it if you agree.
>
> 3.
> Note that after receiving the shutdown signal, an ERROR
> + * is reported if any slots are dropped, invalidated, or inactive. This
> + * measure is taken to prevent the walsender from waiting indefinitely.
> + */
> + if (NeedToWaitForStandby(target_lsn, flushed_lsn, wait_event))
>
> Isn't this part of the comment should be moved inside
> NeedToWaitForStandby()?
Moved.
>
> 4.
> + /*
> + * Update our idea of the currently flushed position only if we are
> + * not waiting for standbys to catch up, otherwise the standby would
> + * have to catch up to a newer WAL location in each cycle.
> + */
> + if (wait_event != WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
> + {
>
> This functionality (in function WalSndWaitForWal()) seems to ensure that we
> first wait for the required WAL to be flushed and then wait for standbys. If true,
> we should cover that point in the comments here or somewhere in the function
> WalSndWaitForWal().
>
> Apart from this, I have made a few modifications in the comments.
Thanks. I have reviewed and merged them.
Here is the V104 patch which addressed above and Peter's comments.
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v104-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch (47.0K, ../../OS0PR01MB5716CD1AB0090155B2ED5C3F945C2@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v104-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch)
download | inline diff:
From 25fba6265323a5cc32c2d682fc1ce9f5bc56e023 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Fri, 23 Feb 2024 08:29:30 +0800
Subject: [PATCH v104 1/2] Allow logical walsenders to wait for the physical
standby
This patch introduces a mechanism to ensure that physical standby servers,
which are potential failover candidates, have received and flushed changes
before making them visible to subscribers. By doing so, it guarantees that
the promoted standby server is not lagging behind the subscribers when a
failover is necessary.
A new parameter named 'standby_slot_names' is introduced. The logical
walsender now guarantees that all local changes are sent and flushed to
the standby servers corresponding to the replication slots specified in
'standby_slot_names' before sending those changes to the subscriber.
Additionally, The SQL functions pg_logical_slot_get_changes,
pg_logical_slot_peek_changes and pg_replication_slot_advance are modified
to wait for the replication slots specified in 'standby_slot_names' to
catch up before returning.
---
doc/src/sgml/config.sgml | 39 ++
doc/src/sgml/func.sgml | 14 +-
doc/src/sgml/logicaldecoding.sgml | 12 +
.../replication/logical/logicalfuncs.c | 12 +
src/backend/replication/logical/slotsync.c | 11 +
src/backend/replication/slot.c | 374 +++++++++++++++++-
src/backend/replication/slotfuncs.c | 12 +
src/backend/replication/walsender.c | 168 +++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 14 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/replication/slot.h | 5 +
src/include/replication/walsender.h | 1 +
src/include/replication/walsender_private.h | 7 +
src/include/utils/guc_hooks.h | 3 +
src/test/recovery/t/006_logical_decoding.pl | 3 +-
.../t/040_standby_failover_slots_sync.pl | 219 ++++++++++
17 files changed, 872 insertions(+), 25 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 43b1a132a2..14552abb22 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4559,6 +4559,45 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Lists the streaming replication standby server slot names that logical
+ WAL sender processes will wait for. Logical WAL sender processes will
+ send decoded changes to plugins only after the specified replication
+ slots confirm receiving WAL. This guarantees that logical replication
+ slots with failover enabled do not consume changes until those changes
+ are received and flushed to corresponding physical standbys. If a
+ logical replication connection is meant to switch to a physical standby
+ after the standby is promoted, the physical replication slot for the
+ standby should be listed here. Note that logical replication will not
+ proceed if the slots specified in the
+ <varname>standby_slot_names</varname> do not exist or are invalidated.
+ Additionally, the replication management functions
+ <link linkend="pg-replication-slot-advance">
+ <function>pg_replication_slot_advance</function></link>,
+ <link linkend="pg-logical-slot-get-changes">
+ <function>pg_logical_slot_get_changes</function></link>, and
+ <link linkend="pg-logical-slot-peek-changes">
+ <function>pg_logical_slot_peek_changes</function></link>,
+ when used with failover enabled logical slots, will block until all
+ physical slots specified in <varname>standby_slot_names</varname> have
+ confirmed WAL receipt.
+ </para>
+ <para>
+ The standbys corresponding to the physical replication slots in
+ <varname>standby_slot_names</varname> must configure
+ <literal>sync_replication_slots = true</literal> so they can receive
+ failover logical slots changes from the primary.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index e5fa82c161..d32edbc2e2 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -28150,7 +28150,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-get-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_get_changes</primary>
</indexterm>
@@ -28173,11 +28173,14 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
the specified value. Note, however, that the actual number of
rows returned may be larger, since this limit is only checked after
adding the rows produced when decoding each new transaction commit.
+ The function may be blocked if the specified slot is a failover enabled
+ slot and <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ is configured.
</para></entry>
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-peek-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_peek_changes</primary>
</indexterm>
@@ -28232,7 +28235,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-replication-slot-advance" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_replication_slot_advance</primary>
</indexterm>
@@ -28248,7 +28251,10 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
the name of the slot and the actual position that it was advanced to.
The updated slot position information is written out at the next
checkpoint if any advancing is done. So in the event of a crash, the
- slot may return to an earlier position.
+ slot may return to an earlier position. The function may be blocked if
+ the specified slot is a failover enabled slot and
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ is configured.
</para></entry>
</row>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 930c0fa8a6..5da3ac7f4c 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -384,6 +384,18 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU
must be enabled on the standby. It is also necessary to specify a valid
<literal>dbname</literal> in the
<link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>.
+ It's highly recommended that the said physical replication slot is named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ list on the primary, to prevent the subscriber from consuming changes
+ faster than the hot standby. Even when correctly configured, some latency
+ is expected when sending changes to logical subscribers due to the waiting
+ on slots named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>.
+ When <varname>standby_slot_names</varname> is utilized, the
+ primary server will not completely shut down until the corresponding
+ standbys, associated with the physical replication slots specified
+ in <varname>standby_slot_names</varname>, have confirmed
+ receiving the WAL up to the latest flushed position on the primary server.
</para>
<para>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b0081d3ce5..f28d6c9015 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -109,6 +109,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
MemoryContext per_query_ctx;
MemoryContext oldcontext;
XLogRecPtr end_of_wal;
+ XLogRecPtr wait_for_wal_lsn;
LogicalDecodingContext *ctx;
ResourceOwner old_resowner = CurrentResourceOwner;
ArrayType *arr;
@@ -228,6 +229,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
NameStr(MyReplicationSlot->data.plugin),
format_procedure(fcinfo->flinfo->fn_oid))));
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to wait_for_wal_lsn.
+ */
+ if (XLogRecPtrIsInvalid(upto_lsn))
+ wait_for_wal_lsn = end_of_wal;
+ else
+ wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
+
+ WaitForStandbyConfirmation(wait_for_wal_lsn);
+
ctx->output_writer_private = p;
/*
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 8ecb85b86a..d1562cf7b3 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -491,6 +491,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
latestFlushPtr = GetStandbyFlushRecPtr(NULL);
if (remote_slot->confirmed_lsn > latestFlushPtr)
{
+ /*
+ * Can get here only if GUC 'standby_slot_names' on the primary server
+ * was not configured correctly.
+ */
ereport(am_slotsync_worker ? LOG : ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("skipping slot synchronization as the received slot sync"
@@ -860,6 +864,13 @@ validate_remote_info(WalReceiverConn *wrconn)
remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull));
Assert(!isnull);
+ /*
+ * Slot sync is currently not supported on a cascading standby. This is
+ * because if we allow it, the primary server needs to wait for all the
+ * cascading standbys, otherwise, logical subscribers can still be ahead
+ * of one of the cascading standbys which we plan to promote. Thus, to
+ * avoid this additional complexity, we restrict it for the time being.
+ */
if (remote_in_recovery)
ereport(ERROR,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 0f173f63a2..77ef083a7f 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -46,13 +46,18 @@
#include "common/string.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/interrupt.h"
#include "replication/slotsync.h"
#include "replication/slot.h"
+#include "replication/walsender_private.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
+#include "utils/memutils.h"
+#include "utils/varlena.h"
/*
* Replication slot on-disk data structure.
@@ -115,10 +120,25 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
/* My backend's replication slot in the shared memory array */
ReplicationSlot *MyReplicationSlot = NULL;
-/* GUC variable */
+/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+/*
+ * This GUC lists streaming replication standby server slot names that
+ * logical WAL sender processes will wait for.
+ */
+char *standby_slot_names;
+
+/* This is parsed and cached list for raw standby_slot_names. */
+static List *standby_slot_names_list = NIL;
+
+/*
+ * Oldest LSN that has been confirmed to be flushed to the standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+static XLogRecPtr standby_slot_oldest_flush_lsn = InvalidXLogRecPtr;
+
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
@@ -2345,3 +2365,355 @@ GetSlotInvalidationCause(const char *conflict_reason)
Assert(found);
return result;
}
+
+/*
+ * A helper function to validate slots specified in GUC standby_slot_names.
+ */
+static bool
+validate_standby_slots(char **newval)
+{
+ char *rawname;
+ List *elemlist;
+ bool ok;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Verify syntax and parse string into a list of identifiers */
+ ok = SplitIdentifierString(rawname, ',', &elemlist);
+
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
+ else if (!ReplicationSlotCtl)
+ {
+ /*
+ * We cannot validate the replication slot if the replication slots'
+ * data has not been initialized. This is ok as we will validate the
+ * specified slot when waiting for them to catch up. See
+ * StandbySlotsHaveCaughtup for details.
+ */
+ }
+ else
+ {
+ /* Check that the specified slots exist and are logical slots */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, elemlist)
+ {
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ GUC_check_errdetail("replication slot \"%s\" does not exist",
+ name);
+ ok = false;
+ break;
+ }
+
+ if (!SlotIsPhysical(slot))
+ {
+ GUC_check_errdetail("\"%s\" is not a physical replication slot",
+ name);
+ ok = false;
+ break;
+ }
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+ }
+
+ pfree(rawname);
+ list_free(elemlist);
+ return ok;
+}
+
+/*
+ * GUC check_hook for standby_slot_names
+ */
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ if ((*newval)[0] == '\0')
+ return true;
+
+ /*
+ * "*" is not accepted as in that case the primary will not be able to
+ * know for which all standbys to wait for. It is inappropriate to block
+ * logical replication for physical slots that either lack associated
+ * standbys or have standbys associated that are not enabled for
+ * replication slot synchronization.
+ */
+ if (strcmp(*newval, "*") == 0)
+ {
+ GUC_check_errdetail("\"*\" is not accepted for standby_slot_names");
+ return false;
+ }
+
+ /* Now verify if the specified slots really exist and have correct type */
+ if (!validate_standby_slots(newval))
+ return false;
+
+ *extra = guc_strdup(ERROR, *newval);
+
+ return true;
+}
+
+/*
+ * GUC assign_hook for standby_slot_names
+ */
+void
+assign_standby_slot_names(const char *newval, void *extra)
+{
+ List *standby_slots;
+ MemoryContext oldcxt;
+ char *standby_slot_names_cpy = extra;
+
+ /*
+ * The standby slots may have changed, so we need to recompute the oldest
+ * LSN.
+ */
+ standby_slot_oldest_flush_lsn = InvalidXLogRecPtr;
+
+ list_free(standby_slot_names_list);
+ standby_slot_names_list = NIL;
+
+ /* No value is specified for standby_slot_names. */
+ if (standby_slot_names_cpy == NULL)
+ return;
+
+ if (!SplitIdentifierString(standby_slot_names_cpy, ',', &standby_slots))
+ {
+ /* This should not happen if GUC checked check_standby_slot_names. */
+ elog(ERROR, "invalid list syntax");
+ }
+
+ /*
+ * Switch to the memory context under which GUC variables are allocated
+ * (GUCMemoryContext).
+ */
+ oldcxt = MemoryContextSwitchTo(GetMemoryChunkContext(standby_slot_names_cpy));
+ standby_slot_names_list = list_copy(standby_slots);
+ MemoryContextSwitchTo(oldcxt);
+}
+
+/*
+ * Return the standby_slot_names_list.
+ *
+ * Note that since we do not support syncing slots to cascading standbys, we
+ * return NIL if we are running in a standby to indicate that no standby slots
+ * need to be waited for.
+ */
+List *
+GetStandbySlotList(void)
+{
+ if (RecoveryInProgress())
+ return NIL;
+
+ return standby_slot_names_list;
+}
+
+/*
+ * Return true if the slots specified in standby_slot_names have caught up to
+ * the given WAL location, false otherwise.
+ *
+ * The elevel parameter specifies the error level used for logging messages
+ * related to slots that do not exist, are invalidated, or are inactive.
+ */
+bool
+StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel)
+{
+ int caught_up_slot_num = 0;
+ XLogRecPtr min_restart_lsn = InvalidXLogRecPtr;
+
+ /*
+ * Don't need to wait for the standbys to catch up if there is no value in
+ * standby_slot_names.
+ */
+ if (standby_slot_names_list == NIL)
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if we are on a standby
+ * server, since we do not support syncing slots to cascading standbys.
+ */
+ if (RecoveryInProgress())
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if they are already
+ * beyond the specified WAL location.
+ */
+ if (!XLogRecPtrIsInvalid(standby_slot_oldest_flush_lsn) &&
+ standby_slot_oldest_flush_lsn >= wait_for_lsn)
+ return true;
+
+ /*
+ * To prevent concurrent slot dropping and creation while filtering the
+ * slots, take the ReplicationSlotControlLock outside of the loop.
+ */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, standby_slot_names_list)
+ {
+ XLogRecPtr restart_lsn;
+ bool invalidated;
+ bool inactive;
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ /*
+ * If a slot name provided in standby_slot_names does not exist,
+ * report a message and exit the loop. A user can specify a slot
+ * name that does not exist just before the server startup. The
+ * GUC check_hook(validate_standby_slots) cannot validate such a
+ * slot during startup as the ReplicationSlotCtl shared memory is
+ * not initialized at that time. It is also possible for a user to
+ * drop the slot in standby_slot_names afterwards.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not exist",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider creating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (SlotIsLogical(slot))
+ {
+ /*
+ * If a logical slot name is provided in standby_slot_names,
+ * report a message and exit the loop. Similar to the non-existent
+ * case, a user can specify a logical slot name in
+ * standby_slot_names before the server startup, or drop an
+ * existing physical slot and recreate a logical slot with the
+ * same name.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot have logical replication slot \"%s\" in parameter %s",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting for correction on \"%s\".",
+ name),
+ errhint("Consider removing logical slot \"%s\" from parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ SpinLockAcquire(&slot->mutex);
+ restart_lsn = slot->data.restart_lsn;
+ invalidated = slot->data.invalidated != RS_INVAL_NONE;
+ inactive = slot->active_pid == 0;
+ SpinLockRelease(&slot->mutex);
+
+ if (invalidated)
+ {
+ /* Specified physical slot has been invalidated */
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("physical slot \"%s\" specified in parameter %s has been invalidated",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider dropping and recreating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (XLogRecPtrIsInvalid(restart_lsn) || restart_lsn < wait_for_lsn)
+ {
+ /* Log a message if no active_pid for this physical slot */
+ if (inactive)
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not have active_pid",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider starting standby associated with \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+
+ /* Continue if the current slot hasn't caught up. */
+ break;
+ }
+
+ Assert(restart_lsn >= wait_for_lsn);
+
+ if (XLogRecPtrIsInvalid(min_restart_lsn) ||
+ min_restart_lsn > restart_lsn)
+ min_restart_lsn = restart_lsn;
+
+ caught_up_slot_num++;
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /*
+ * Return false if not all the standbys have caught up to the specified
+ * WAL location.
+ */
+ if (caught_up_slot_num != list_length(standby_slot_names_list))
+ return false;
+
+ Assert(XLogRecPtrIsInvalid(standby_slot_oldest_flush_lsn) ||
+ min_restart_lsn >= standby_slot_oldest_flush_lsn);
+
+ standby_slot_oldest_flush_lsn = min_restart_lsn;
+
+ return true;
+}
+
+/*
+ * Wait for physical standbys to confirm receiving the given lsn.
+ *
+ * Used by logical decoding SQL functions that acquired failover enabled slot.
+ * It waits for physical standbys corresponding to the physical slots specified
+ * in the standby_slot_names GUC.
+ */
+void
+WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
+{
+ /*
+ * Don't need to wait for the standby to catch up if the current acquired
+ * slot is not a failover enabled slot, or there is no value in
+ * standby_slot_names.
+ */
+ if (!MyReplicationSlot->data.failover || standby_slot_names_list == NIL)
+ return;
+
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+
+ for (;;)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /* Exit if done waiting for every slot. */
+ if (StandbySlotsHaveCaughtup(wait_for_lsn, WARNING))
+ break;
+
+ /*
+ * Wait for the slots in the standby_slot_names to catch up, but use a
+ * timeout (1s) so we can also check if the standby_slot_names has
+ * been changed.
+ */
+ ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000,
+ WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION);
+ }
+
+ ConditionVariableCancelSleep();
+}
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 768a304723..d864fe4133 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -464,6 +464,12 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* crash, but this makes the data consistent after a clean shutdown.
*/
ReplicationSlotMarkDirty();
+
+ /*
+ * Wake up logical walsenders holding failover enabled slots after
+ * updating the restart_lsn of the physical slot.
+ */
+ PhysicalWakeupLogicalWalSnd();
}
return retlsn;
@@ -504,6 +510,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto)
.segment_close = wal_segment_close),
NULL, NULL, NULL);
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to moveto lsn.
+ */
+ WaitForStandbyConfirmation(moveto);
+
/*
* Start reading at the slot's restart_lsn, which we know to point to
* a valid record.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 13bc3e0aee..a795a4af76 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1728,26 +1728,121 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
ProcessPendingWrites();
}
+/*
+ * Wake up the logical walsender processes with failover enabled slots if the
+ * currently acquired physical slot is specified in standby_slot_names GUC.
+ */
+void
+PhysicalWakeupLogicalWalSnd(void)
+{
+ List *standby_slots;
+
+ Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot));
+
+ standby_slots = GetStandbySlotList();
+
+ foreach_ptr(char, name, standby_slots)
+ {
+ if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0)
+ {
+ ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv);
+ return;
+ }
+ }
+}
+
+/*
+ * Returns true if not all standbys have caught up to the flushed position
+ * (flushed_lsn) when the current acquired slot is a failover enabled logical
+ * slot and we are streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForStandby(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
+{
+ int elevel = got_STOPPING ? ERROR : WARNING;
+ bool failover_slot;
+
+ failover_slot = (replication_active && MyReplicationSlot->data.failover);
+
+ /*
+ * Note that after receiving the shutdown signal, an ERROR is reported if
+ * any slots are dropped, invalidated, or inactive. This measure is taken
+ * to prevent the walsender from waiting indefinitely.
+ */
+ if (failover_slot && !StandbySlotsHaveCaughtup(flushed_lsn, elevel))
+ {
+ *wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
+ return true;
+ }
+
+ *wait_event = 0;
+ return false;
+}
+
+/*
+ * Returns true if we need to wait for WALs to be flushed to disk, or if not
+ * all standbys have caught up to the flushed position (flushed_lsn) when the
+ * current acquired slot is a failover enabled logical slot and we are
+ * streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
+{
+ /* Check if we need to wait for WALs to be flushed to disk. */
+ if (target_lsn > flushed_lsn)
+ {
+ *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
+ return true;
+ }
+
+ /*
+ * Check if the standby slots have caught up to the flushed position. It
+ * is good to wait up to flushed position and then let it send the changes
+ * to logical subscribers one by one which are already covered in flushed
+ * position without needing to wait on every change for standby
+ * confirmation.
+ */
+ if (NeedToWaitForStandby(target_lsn, flushed_lsn, wait_event))
+ return true;
+
+ *wait_event = 0;
+ return false;
+}
+
/*
* Wait till WAL < loc is flushed to disk so it can be safely sent to client.
*
- * Returns end LSN of flushed WAL. Normally this will be >= loc, but
- * if we detect a shutdown request (either from postmaster or client)
- * we will return early, so caller must always check.
+ * If the walsender holds a logical slot that has enabled failover, we also
+ * wait for all the specified streaming replication standby servers to
+ * confirm receipt of WAL up to RecentFlushPtr.
+ *
+ * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we
+ * detect a shutdown request (either from postmaster or client) we will return
+ * early, so caller must always check.
*/
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
int wakeEvents;
+ uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
- * have enough WAL available. This is particularly interesting if we're
- * far behind.
+ * have enough WAL available and all the standby servers have confirmed
+ * receipt of WAL up to RecentFlushPtr. This is particularly interesting
+ * if we're far behind.
*/
- if (RecentFlushPtr != InvalidXLogRecPtr &&
- loc <= RecentFlushPtr)
+ if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
return RecentFlushPtr;
/* Get a more recent flush pointer. */
@@ -1758,6 +1853,7 @@ WalSndWaitForWal(XLogRecPtr loc)
for (;;)
{
+ bool wait_for_standby_at_stop = false;
long sleeptime;
/* Clear any already-pending wakeups */
@@ -1784,21 +1880,38 @@ WalSndWaitForWal(XLogRecPtr loc)
if (got_STOPPING)
XLogBackgroundFlush();
- /* Update our idea of the currently flushed position. */
- if (!RecoveryInProgress())
- RecentFlushPtr = GetFlushRecPtr(NULL);
- else
- RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * Within the loop, we wait for the necessary WALs to be flushed to
+ * disk first, followed by waiting for standbys to catch up if there
+ * are enought WALs or upon receiving the shutdown signal. To avoid
+ * the scenario where standbys need to catch up to a newer WAL
+ * location in each iteration, we update our idea of the currently
+ * flushed position only if we are not waiting for standbys to catch
+ * up.
+ */
+ if (wait_event != WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ {
+ if (!RecoveryInProgress())
+ RecentFlushPtr = GetFlushRecPtr(NULL);
+ else
+ RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ }
/*
- * If postmaster asked us to stop, don't wait anymore.
+ * If postmaster asked us to stop and the standby slots have caught up
+ * to the flushed position, don't wait anymore.
*
* It's important to do this check after the recomputation of
* RecentFlushPtr, so we can send all remaining data before shutting
* down.
*/
if (got_STOPPING)
- break;
+ {
+ if (NeedToWaitForStandby(loc, RecentFlushPtr, &wait_event))
+ wait_for_standby_at_stop = true;
+ else
+ break;
+ }
/*
* We only send regular messages to the client for full decoded
@@ -1813,11 +1926,18 @@ WalSndWaitForWal(XLogRecPtr loc)
!waiting_for_ping_response)
WalSndKeepalive(false, InvalidXLogRecPtr);
- /* check whether we're done */
- if (loc <= RecentFlushPtr)
+ /*
+ * Exit the loop if already caught up and doesn't need to wait for
+ * standby slots.
+ */
+ if (!wait_for_standby_at_stop &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
break;
- /* Waiting for new WAL. Since we need to wait, we're now caught up. */
+ /*
+ * Waiting for new WAL or waiting for standbys to catch up. Since we
+ * need to wait, we're now caught up.
+ */
WalSndCaughtUp = true;
/*
@@ -1855,7 +1975,9 @@ WalSndWaitForWal(XLogRecPtr loc)
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL);
+ Assert(wait_event != 0);
+
+ WalSndWait(wakeEvents, sleeptime, wait_event);
}
/* reactivate latch so WalSndLoop knows to continue */
@@ -2265,6 +2387,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn)
{
ReplicationSlotMarkDirty();
ReplicationSlotsComputeRequiredLSN();
+ PhysicalWakeupLogicalWalSnd();
}
/*
@@ -3538,6 +3661,7 @@ WalSndShmemInit(void)
ConditionVariableInit(&WalSndCtl->wal_flush_cv);
ConditionVariableInit(&WalSndCtl->wal_replay_cv);
+ ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv);
}
}
@@ -3607,8 +3731,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event)
*
* And, we use separate shared memory CVs for physical and logical
* walsenders for selective wake ups, see WalSndWakeup() for more details.
+ *
+ * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV
+ * until awakened by physical walsenders after the walreceiver confirms
+ * the receipt of the LSN.
*/
- if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
+ if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+ else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv);
else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index ec2f31f82a..c08e00d1d6 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP
LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server."
LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server."
SSL_OPEN_SERVER "Waiting for SSL while attempting connection."
+WAIT_FOR_STANDBY_CONFIRMATION "Waiting for WAL to be received and flushed by the physical standby."
WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process."
WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 93ded31ed9..6741b2a10c 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4670,6 +4670,20 @@ struct config_string ConfigureNamesString[] =
check_debug_io_direct, assign_debug_io_direct, NULL
},
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("Lists streaming replication standby server slot "
+ "names that logical WAL sender processes will wait for."),
+ gettext_noop("Logical WAL sender processes will send decoded "
+ "changes to plugins only after the specified "
+ "replication slots confirm receiving WAL."),
+ GUC_LIST_INPUT
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, assign_standby_slot_names, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index edcc0282b2..2244ee52f7 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -343,6 +343,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsender processes will wait for
# - Standby Servers -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index acbf567150..61a6b97952 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -226,6 +226,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
+extern PGDLLIMPORT char *standby_slot_names;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -274,4 +275,8 @@ extern void CheckSlotPermissions(void);
extern ReplicationSlotInvalidationCause
GetSlotInvalidationCause(const char *conflict_reason);
+extern List *GetStandbySlotList(void);
+extern bool StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel);
+extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn);
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index 0c3996e926..f2d8297f01 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -39,6 +39,7 @@ extern void InitWalSender(void);
extern bool exec_replication_command(const char *cmd_string);
extern void WalSndErrorCleanup(void);
extern void WalSndResourceCleanup(bool isCommit);
+extern void PhysicalWakeupLogicalWalSnd(void);
extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
extern void WalSndSignals(void);
extern Size WalSndShmemSize(void);
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 3113e9ea47..3c134c8edf 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -113,6 +113,13 @@ typedef struct
ConditionVariable wal_flush_cv;
ConditionVariable wal_replay_cv;
+ /*
+ * Used by physical walsenders holding slots specified in
+ * standby_slot_names to wake up logical walsenders holding failover
+ * enabled slots when a walreceiver confirms the receipt of LSN.
+ */
+ ConditionVariable wal_confirm_rcv_cv;
+
WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER];
} WalSndCtlData;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index c8a7aa9a11..d64dc5fcdb 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra,
extern void assign_wal_consistency_checking(const char *newval, void *extra);
extern bool check_wal_segment_size(int *newval, void **extra, GucSource source);
extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
+extern bool check_standby_slot_names(char **newval, void **extra,
+ GucSource source);
+extern void assign_standby_slot_names(const char *newval, void *extra);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl
index 5c7b4ca5e3..b95d95c06f 100644
--- a/src/test/recovery/t/006_logical_decoding.pl
+++ b/src/test/recovery/t/006_logical_decoding.pl
@@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'},
undef, 'logical slot was actually dropped with DB');
# Test logical slot advancing and its durability.
+# Passing failover=true (last arg) should not have any impact on advancing.
my $logical_slot = 'logical_slot';
$node_primary->safe_psql('postgres',
- "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);"
+ "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);"
);
$node_primary->psql(
'postgres', "
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 021c58f621..4abed7c335 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -504,11 +504,230 @@ ok( $standby1->poll_query_until(
"SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot' AND synced AND NOT temporary;"),
'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby');
+##################################################
+# Test that logical failover replication slots wait for the specified
+# physical replication slots to receive the changes first. It uses the
+# following set up:
+#
+# (physical standbys)
+# | ----> standby1 (primary_slot_name = sb1_slot)
+# | ----> standby2 (primary_slot_name = sb2_slot)
+# primary ----- |
+# (logical replication)
+# | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
+# | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
+#
+# standby_slot_names = 'sb1_slot'
+#
+# The setup is configured in such a way that the logical slot of subscriber1 is
+# enabled for failover, and thus the subscriber1 will wait for the physical
+# slot of standby1(sb1_slot) to catch up before receiving the decoded changes.
+##################################################
+
+$backup_name = 'backup3';
+
+$primary->psql('postgres',
+ q{SELECT pg_create_physical_replication_slot('sb2_slot');});
+
+$primary->backup($backup_name);
+
+# Create another standby
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup(
+ $primary, $backup_name,
+ has_streaming => 1,
+ has_restoring => 1);
+$standby2->append_conf(
+ 'postgresql.conf', qq(
+primary_slot_name = 'sb2_slot'
+));
+$standby2->start;
+$primary->wait_for_replay_catchup($standby2);
+
+# Configure primary to disallow any logical slots that have enabled failover
+# from getting ahead of the specified physical replication slot (sb1_slot).
+$primary->append_conf(
+ 'postgresql.conf', qq(
+standby_slot_names = 'sb1_slot'
+));
+$primary->reload;
+
+# Create another subscriber node without enabling failover, wait for sync to
+# complete
+my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$subscriber2->init;
+$subscriber2->start;
+$subscriber2->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE tab_int (a int PRIMARY KEY);
+ CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot);
+]);
+
+$subscriber2->wait_for_subscription_sync;
+
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+my $offset = -s $primary->logfile;
+
+# Stop the standby associated with the specified physical replication slot
+# (sb1_slot) so that the logical replication slot (lsub1_slot) won't receive
+# changes until the standby comes up.
+$standby1->stop;
+
+# Create some data on the primary
+my $primary_row_count = 20;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(11, $primary_row_count);");
+
+# Wait until the standby2 that's still running gets the data from the primary
+$primary->wait_for_replay_catchup($standby2);
+$result = $standby2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby2 gets data from primary");
+
+# Wait for regress_mysub2 to get the data from the primary. This subscription
+# was not enabled for failover so it gets the data without waiting for any
+# standbys.
+$primary->wait_for_catchup('regress_mysub2');
+$result = $subscriber2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "subscriber2 gets data from primary");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# The regress_mysub1 was enabled for failover so it doesn't get the data from
+# primary and keeps waiting for the standby specified in standby_slot_names
+# (sb1_slot aka standby1).
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) <> $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data from primary until standby1 acknowledges changes"
+);
+
+# Start the standby specified in standby_slot_names (sb1_slot aka standby1) and
+# wait for it to catch up with the primary.
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+$result = $standby1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby1 gets data from primary");
+
+# Now that the standby specified in standby_slot_names is up and running, the
+# primary can send the decoded changes to the subscription enabled for failover
+# (i.e. regress_mysub1). While the standby was down, regress_mysub1 didn't
+# receive any data from the primary. i.e. the primary didn't allow it to go
+# ahead of standby.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 acknowledges changes");
+
+##################################################
+# Verify that when using pg_logical_slot_get_changes to consume changes from a
+# logical slot with failover enabled, it will also wait for the slots specified
+# in standby_slot_names to catch up.
+##################################################
+
+# Stop the standby associated with the specified physical replication slot so
+# that the logical replication slot won't receive changes until the standby
+# slot's restart_lsn is advanced or the slot is removed from the
+# standby_slot_names list.
+$primary->safe_psql('postgres', "TRUNCATE tab_int;");
+$primary->wait_for_catchup('regress_mysub1');
+$standby1->stop;
+
+# Create a logical 'test_decoding' replication slot with failover enabled
+$primary->safe_psql('postgres',
+ "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);"
+);
+
+my $back_q = $primary->background_psql(
+ 'postgres',
+ on_error_stop => 0,
+ timeout => $PostgreSQL::Test::Utils::timeout_default);
+
+# Try and get changes from the logical slot with failover enabled.
+$offset = -s $primary->logfile;
+$back_q->query_until(qr//,
+ "SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);\n");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the function
+# pg_logical_slot_get_changes should now return, and the session can be
+# stopped.
+$back_q->quit;
+
+$primary->safe_psql('postgres',
+ "SELECT pg_drop_replication_slot('test_slot');"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
+##################################################
+# Test that logical replication will wait for the user-created inactive
+# physical slot to catch up until we remove the slot from standby_slot_names.
+##################################################
+
+$offset = -s $primary->logfile;
+
+# Create some data on the primary
+$primary_row_count = 10;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# The regress_mysub1 doesn't get the data from primary because the specified
+# standby slot (sb1_slot) in standby_slot_names is inactive.
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data as the sb1_slot doesn't catch up");
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the primary server should now
+# send the decoded changes to the subscription.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list"
+);
+
##################################################
# Promote the standby1 to primary. Confirm that:
# a) the slot 'lsub1_slot' is retained on the new primary
# b) logical replication for regress_mysub1 is resumed successfully after failover
##################################################
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+
$standby1->promote;
# Update subscription with the new primary's connection info
--
2.30.0.windows.2
[application/octet-stream] v104-0002-Document-the-steps-to-check-if-the-standby-is-r.patch (7.0K, ../../OS0PR01MB5716CD1AB0090155B2ED5C3F945C2@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v104-0002-Document-the-steps-to-check-if-the-standby-is-r.patch)
download | inline diff:
From 9ec25c16f3efbc7e51694d2132c11021f5a6b640 Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Fri, 19 Jan 2024 11:04:16 +0530
Subject: [PATCH v104 2/2] Document the steps to check if the standby is ready
for failover
---
doc/src/sgml/high-availability.sgml | 9 ++
doc/src/sgml/logical-replication.sgml | 136 ++++++++++++++++++++++++++
2 files changed, 145 insertions(+)
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 236c0af65f..36215aa68c 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -1487,6 +1487,15 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)'
Written administration procedures are advised.
</para>
+ <para>
+ If you have opted for synchronization of logical slots (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>),
+ then before switching to the standby server, it is recommended to check
+ if the logical slots synchronized on the standby server are ready
+ for failover. This can be done by following the steps described in
+ <xref linkend="logical-replication-failover"/>.
+ </para>
+
<para>
To trigger failover of a log-shipping standby server, run
<command>pg_ctl promote</command> or call <function>pg_promote()</function>.
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ec2130669e..be59d306a1 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -687,6 +687,142 @@ ALTER SUBSCRIPTION
</sect1>
+ <sect1 id="logical-replication-failover">
+ <title>Logical Replication Failover</title>
+
+ <para>
+ When the publisher server is the primary server of a streaming replication,
+ the logical slots on that primary server can be synchronized to the standby
+ server by specifying <literal>failover = true</literal> when creating
+ subscriptions for those publications. Enabling failover ensures a seamless
+ transition of those subscriptions after the standby is promoted. They can
+ continue subscribing to publications now on the new primary server without
+ any data loss.
+ </para>
+
+ <para>
+ Because the slot synchronization logic copies asynchronously, it is
+ necessary to confirm that replication slots have been synced to the standby
+ server before the failover happens. Furthermore, to ensure a successful
+ failover, the standby server must not be lagging behind the subscriber. It
+ is highly recommended to use
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ to prevent the subscriber from consuming changes faster than the hot standby.
+ To confirm that the standby server is indeed ready for failover, follow
+ these 2 steps:
+ </para>
+
+ <procedure>
+ <step performance="required">
+ <para>
+ Confirm that all the necessary logical replication slots have been synced to
+ the standby server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node, use the following SQL to identify
+ which slots should be synced to the standby that we plan to promote.
+<programlisting>
+test_sub=# SELECT
+ array_agg(slotname) AS slots
+ FROM
+ ((
+ SELECT r.srsubid AS subid, CONCAT('pg_', srsubid, '_sync_', srrelid, '_', ctl.system_identifier) AS slotname
+ FROM pg_control_system() ctl, pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate = 'f' AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT s.oid AS subid, s.subslotname as slotname
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ slots
+-------
+ {sub1,sub2,sub3}
+(1 row)
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, check that the logical replication slots identified above exist on
+ the standby server and are ready for failover.
+<programlisting>
+test_standby=# SELECT slot_name, (synced AND NOT temporary AND conflict_reason IS NULL) AS failover_ready
+ FROM pg_replication_slots
+ WHERE slot_name IN ('sub1','sub2','sub3');
+ slot_name | failover_ready
+-------------+----------------
+ sub1 | t
+ sub2 | t
+ sub3 | t
+(3 rows)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+
+ <step performance="required">
+ <para>
+ Confirm that the standby server is not lagging behind the subscribers.
+ This step can be skipped if
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ has been correctly configured. If standby_slot_names is not configured
+ correctly, it is highly recommended to run this step after the primary
+ server is down, otherwise the results of the query may vary at different
+ points of time due to the ongoing replication on the logical subscribers
+ from the primary server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node check the last replayed WAL.
+ This step needs to be run on the database(s) that includes the failover
+ enabled subscription(s), to find the last replayed WAL on each database.
+<programlisting>
+test_sub=# SELECT
+ MAX(remote_lsn) AS remote_lsn_on_subscriber
+ FROM
+ ((
+ SELECT (CASE WHEN r.srsubstate = 'f' THEN pg_replication_origin_progress(CONCAT('pg_', r.srsubid, '_', r.srrelid), false)
+ WHEN r.srsubstate IN ('s', 'r') THEN r.srsublsn END) AS remote_lsn
+ FROM pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate IN ('f', 's', 'r') AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT pg_replication_origin_progress(CONCAT('pg_', s.oid), false) AS remote_lsn
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ remote_lsn_on_subscriber
+--------------------------
+ 0/3000388
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, on the standby server check that the last-received WAL location
+ is ahead of the replayed WAL location(s) on the subscriber identified
+ above. If the above SQL result was NULL, it means the subscriber has not
+ yet replayed any WAL, so the standby server must be ahead of the
+ subscriber, and this step can be skipped.
+<programlisting>
+test_standby=# SELECT pg_last_wal_receive_lsn() >= '0/3000388'::pg_lsn AS failover_ready;
+ failover_ready
+----------------
+ t
+(1 row)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+ </procedure>
+
+ <para>
+ If the result (<literal>failover_ready</literal>) of both above steps is
+ true, existing subscriptions will be able to continue without data loss.
+ </para>
+
+ </sect1>
+
<sect1 id="logical-replication-row-filter">
<title>Row Filters</title>
--
2.30.0.windows.2
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-04 01:27 Peter Smith <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Peter Smith @ 2024-03-04 01:27 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Sun, Mar 3, 2024 at 2:56 PM Amit Kapila <[email protected]> wrote:
>
> On Sun, Mar 3, 2024 at 5:17 AM Peter Smith <[email protected]> wrote:
> >
...
> > 9. NeedToWaitForWal
> >
> > + /*
> > + * Check if the standby slots have caught up to the flushed position. It
> > + * is good to wait up to flushed position and then let it send the changes
> > + * to logical subscribers one by one which are already covered in flushed
> > + * position without needing to wait on every change for standby
> > + * confirmation. Note that after receiving the shutdown signal, an ERROR
> > + * is reported if any slots are dropped, invalidated, or inactive. This
> > + * measure is taken to prevent the walsender from waiting indefinitely.
> > + */
> > + if (NeedToWaitForStandby(target_lsn, flushed_lsn, wait_event))
> > + return true;
> >
> > I felt it was confusing things for this function to also call to the
> > other one -- it seems an overlapping/muddling of the purpose of these.
> > I think it will be easier to understand if the calling code just calls
> > to one or both of these functions as required.
> >
>
> I felt otherwise because the caller has to call these functions at
> more than one place which makes the caller's code difficult to follow.
> It is better to encapsulate the computation of wait_event.
>
You may have misinterpreted my review comment because I didn't say
anything about changing the encapsulation of the computation of the
wait_event.
I only wrote it is better IMO for the functions to stick to just one
job each according to their function name. E.g.:
- NeedToWaitForStandby should *only* check if we need to wait for standbys
- NeedToWaitForWal should *only* check if we need to wait for WAL
flush; i.e. this shouldn't be also calling NeedToWaitForStandby().
Also, AFAICT the caller changes should not be difficult. Indeed, these
changes will make the code aligned properly with what the comments are
saying:
BEFORE
/*
* Fast path to avoid acquiring the spinlock in case we already know we
* have enough WAL available and all the standby servers have confirmed
* receipt of WAL up to RecentFlushPtr. This is particularly interesting
* if we're far behind.
*/
if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
!NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
return RecentFlushPtr;
SUGGESTED
...
if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
!NeedToWaitForWal(loc, RecentFlushPtr, &wait_event) &&
!NeedToWaitForStandby(loc, RecentFlushPtr, &wait_event)
return RecentFlushPtr;
~~~
BEFORE
/*
* Exit the loop if already caught up and doesn't need to wait for
* standby slots.
*/
if (!wait_for_standby_at_stop &&
!NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
break;
SUGGESTED
...
if (!wait_for_standby_at_stop &&
!NeedToWaitForWal(loc, RecentFlushPtr, &wait_event) &&
!NeedToWaitForStandby(loc, RecentFlushPtr, &wait_event))
break;
----------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-04 01:55 Peter Smith <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 2 replies; 68+ messages in thread
From: Peter Smith @ 2024-03-04 01:55 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Sun, Mar 3, 2024 at 6:51 PM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> On Sunday, March 3, 2024 7:47 AM Peter Smith <[email protected]> wrote:
> >
> > 3.
> > + <note>
> > + <para>
> > + Value <literal>*</literal> is not accepted as it is inappropriate to
> > + block logical replication for physical slots that either lack
> > + associated standbys or have standbys associated that are not
> > enabled
> > + for replication slot synchronization. (see
> > + <xref
> > linkend="logicaldecoding-replication-slots-synchronization"/>).
> > + </para>
> > + </note>
> >
> > Why does the document need to provide an excuse/reason for the rule?
> > You could just say something like:
> >
> > SUGGESTION
> > The slots must be named explicitly. For example, specifying wildcard values like
> > <literal>*</literal> is not permitted.
>
> As suggested by Amit, I moved this to code comments.
Was the total removal of this note deliberate? I only suggested
removing the *reason* for the rule, not the entire rule. Otherwise,
the user won't know to avoid doing this until they try it and get an
error.
> >
> > 9. NeedToWaitForWal
> >
> > + /*
> > + * Check if the standby slots have caught up to the flushed position.
> > + It
> > + * is good to wait up to flushed position and then let it send the
> > + changes
> > + * to logical subscribers one by one which are already covered in
> > + flushed
> > + * position without needing to wait on every change for standby
> > + * confirmation. Note that after receiving the shutdown signal, an
> > + ERROR
> > + * is reported if any slots are dropped, invalidated, or inactive. This
> > + * measure is taken to prevent the walsender from waiting indefinitely.
> > + */
> > + if (NeedToWaitForStandby(target_lsn, flushed_lsn, wait_event)) return
> > + true;
> >
> > I felt it was confusing things for this function to also call to the other one -- it
> > seems an overlapping/muddling of the purpose of these.
> > I think it will be easier to understand if the calling code just calls to one or both
> > of these functions as required.
>
> Same as Amit, I didn't change this.
AFAICT my previous review comment was misinterpreted. Please see [1]
for more details.
~~~~
Here are some more review comments for v104-00001
======
Commit message
1.
Additionally, The SQL functions pg_logical_slot_get_changes,
pg_logical_slot_peek_changes and pg_replication_slot_advance are modified
to wait for the replication slots specified in 'standby_slot_names' to
catch up before returning.
~
Maybe that should be expressed using similar wording as the docs...
SUGGESTION
Additionally, The SQL functions ... are modified. Now, when used with
failover-enabled logical slots, these functions will block until all
physical slots specified in 'standby_slot_names' have confirmed WAL
receipt.
======
doc/src/sgml/config.sgml
2.
+ <function>pg_logical_slot_peek_changes</function></link>,
+ when used with failover enabled logical slots, will block until all
+ physical slots specified in <varname>standby_slot_names</varname> have
+ confirmed WAL receipt.
/failover enabled logical slots/failover-enabled logical slots/
======
doc/src/sgml/func.sgml
3.
+ The function may be blocked if the specified slot is a failover enabled
+ slot and <link
linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ is configured.
</para></entry>
/a failover enabled slot//a failover-enabled slot/
~~~
4.
+ slot may return to an earlier position. The function may be blocked if
+ the specified slot is a failover enabled slot and
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ is configured.
/a failover enabled slot//a failover-enabled slot/
======
src/backend/replication/slot.c
5.
+/*
+ * Wait for physical standbys to confirm receiving the given lsn.
+ *
+ * Used by logical decoding SQL functions that acquired failover enabled slot.
+ * It waits for physical standbys corresponding to the physical slots specified
+ * in the standby_slot_names GUC.
+ */
+void
+WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
/failover enabled slot/failover-enabled slot/
~~~
6.
+ /*
+ * Don't need to wait for the standby to catch up if the current acquired
+ * slot is not a failover enabled slot, or there is no value in
+ * standby_slot_names.
+ */
/failover enabled slot/failover-enabled slot/
======
src/backend/replication/slotfuncs.c
7.
+
+ /*
+ * Wake up logical walsenders holding failover enabled slots after
+ * updating the restart_lsn of the physical slot.
+ */
+ PhysicalWakeupLogicalWalSnd();
/failover enabled slots/failover-enabled slots/
======
src/backend/replication/walsender.c
8.
+/*
+ * Wake up the logical walsender processes with failover enabled slots if the
+ * currently acquired physical slot is specified in standby_slot_names GUC.
+ */
+void
+PhysicalWakeupLogicalWalSnd(void)
/failover enabled slots/failover-enabled slots/
9.
+/*
+ * Returns true if not all standbys have caught up to the flushed position
+ * (flushed_lsn) when the current acquired slot is a failover enabled logical
+ * slot and we are streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForStandby(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
9a.
/failover enabled logical slot/failover-enabled logical slot/
~
9b.
Probably that function name should be plural.
/NeedToWaitForStandby/NeedToWaitForStandbys/
~~~
10.
+/*
+ * Returns true if we need to wait for WALs to be flushed to disk, or if not
+ * all standbys have caught up to the flushed position (flushed_lsn) when the
+ * current acquired slot is a failover enabled logical slot and we are
+ * streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
/failover enabled logical slot/failover-enabled logical slot/
~~~
11. WalSndWaitForWal
+ /*
+ * Within the loop, we wait for the necessary WALs to be flushed to
+ * disk first, followed by waiting for standbys to catch up if there
+ * are enought WALs or upon receiving the shutdown signal. To avoid
+ * the scenario where standbys need to catch up to a newer WAL
+ * location in each iteration, we update our idea of the currently
+ * flushed position only if we are not waiting for standbys to catch
+ * up.
+ */
typo
/enought/enough/
----------
[1] https://www.postgresql.org/message-id/CAHut%2BPteoyDki-XdygDgoaZJLmasutzRquQepYx0raNs0RSMvg%40mail.g...
Kind Regards,
Peter Smith.
Fujitsu Austalia
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-04 03:38 Amit Kapila <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Amit Kapila @ 2024-03-04 03:38 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Mon, Mar 4, 2024 at 6:57 AM Peter Smith <[email protected]> wrote:
>
> On Sun, Mar 3, 2024 at 2:56 PM Amit Kapila <[email protected]> wrote:
> >
> > On Sun, Mar 3, 2024 at 5:17 AM Peter Smith <[email protected]> wrote:
> > >
> ...
> > > 9. NeedToWaitForWal
> > >
> > > + /*
> > > + * Check if the standby slots have caught up to the flushed position. It
> > > + * is good to wait up to flushed position and then let it send the changes
> > > + * to logical subscribers one by one which are already covered in flushed
> > > + * position without needing to wait on every change for standby
> > > + * confirmation. Note that after receiving the shutdown signal, an ERROR
> > > + * is reported if any slots are dropped, invalidated, or inactive. This
> > > + * measure is taken to prevent the walsender from waiting indefinitely.
> > > + */
> > > + if (NeedToWaitForStandby(target_lsn, flushed_lsn, wait_event))
> > > + return true;
> > >
> > > I felt it was confusing things for this function to also call to the
> > > other one -- it seems an overlapping/muddling of the purpose of these.
> > > I think it will be easier to understand if the calling code just calls
> > > to one or both of these functions as required.
> > >
> >
> > I felt otherwise because the caller has to call these functions at
> > more than one place which makes the caller's code difficult to follow.
> > It is better to encapsulate the computation of wait_event.
> >
>
> You may have misinterpreted my review comment because I didn't say
> anything about changing the encapsulation of the computation of the
> wait_event.
>
No, I have understood it in the same way as you have outlined in this
email and liked the way the current patch has it.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-04 03:49 Amit Kapila <[email protected]>
parent: Peter Smith <[email protected]>
1 sibling, 1 reply; 68+ messages in thread
From: Amit Kapila @ 2024-03-04 03:49 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Mon, Mar 4, 2024 at 7:25 AM Peter Smith <[email protected]> wrote:
>
>
> ======
> doc/src/sgml/config.sgml
>
> 2.
> + <function>pg_logical_slot_peek_changes</function></link>,
> + when used with failover enabled logical slots, will block until all
> + physical slots specified in <varname>standby_slot_names</varname> have
> + confirmed WAL receipt.
>
> /failover enabled logical slots/failover-enabled logical slots/
>
How about just saying logical failover slots at this and other places?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-04 03:57 Peter Smith <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 0 replies; 68+ messages in thread
From: Peter Smith @ 2024-03-04 03:57 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Mon, Mar 4, 2024 at 2:49 PM Amit Kapila <[email protected]> wrote:
>
> On Mon, Mar 4, 2024 at 7:25 AM Peter Smith <[email protected]> wrote:
> >
> >
> > ======
> > doc/src/sgml/config.sgml
> >
> > 2.
> > + <function>pg_logical_slot_peek_changes</function></link>,
> > + when used with failover enabled logical slots, will block until all
> > + physical slots specified in <varname>standby_slot_names</varname> have
> > + confirmed WAL receipt.
> >
> > /failover enabled logical slots/failover-enabled logical slots/
> >
>
> How about just saying logical failover slots at this and other places?
>
Yes, that wording works too.
----------
Kind Regards,
Peter Smith.
Fujitsu Australia.
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-04 04:05 Peter Smith <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Peter Smith @ 2024-03-04 04:05 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Mon, Mar 4, 2024 at 2:38 PM Amit Kapila <[email protected]> wrote:
>
> On Mon, Mar 4, 2024 at 6:57 AM Peter Smith <[email protected]> wrote:
> >
> > On Sun, Mar 3, 2024 at 2:56 PM Amit Kapila <[email protected]> wrote:
> > >
> > > On Sun, Mar 3, 2024 at 5:17 AM Peter Smith <[email protected]> wrote:
> > > >
> > ...
> > > > 9. NeedToWaitForWal
> > > >
> > > > + /*
> > > > + * Check if the standby slots have caught up to the flushed position. It
> > > > + * is good to wait up to flushed position and then let it send the changes
> > > > + * to logical subscribers one by one which are already covered in flushed
> > > > + * position without needing to wait on every change for standby
> > > > + * confirmation. Note that after receiving the shutdown signal, an ERROR
> > > > + * is reported if any slots are dropped, invalidated, or inactive. This
> > > > + * measure is taken to prevent the walsender from waiting indefinitely.
> > > > + */
> > > > + if (NeedToWaitForStandby(target_lsn, flushed_lsn, wait_event))
> > > > + return true;
> > > >
> > > > I felt it was confusing things for this function to also call to the
> > > > other one -- it seems an overlapping/muddling of the purpose of these.
> > > > I think it will be easier to understand if the calling code just calls
> > > > to one or both of these functions as required.
> > > >
> > >
> > > I felt otherwise because the caller has to call these functions at
> > > more than one place which makes the caller's code difficult to follow.
> > > It is better to encapsulate the computation of wait_event.
> > >
> >
> > You may have misinterpreted my review comment because I didn't say
> > anything about changing the encapsulation of the computation of the
> > wait_event.
> >
>
> No, I have understood it in the same way as you have outlined in this
> email and liked the way the current patch has it.
>
OK, if the code will remain as-is wouldn't it be better to anyway
change the function name to indicate what it really does?
e.g. NeedToWaitForWal --> NeedToWaitForWalFlushOrStandbys
----------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-04 08:57 Bertrand Drouvot <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Bertrand Drouvot @ 2024-03-04 08:57 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Peter Smith <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
Hi,
On Thu, Feb 29, 2024 at 03:38:59PM +0530, Amit Kapila wrote:
> On Thu, Feb 29, 2024 at 9:13 AM Peter Smith <[email protected]> wrote:
> >
> > On Tue, Feb 27, 2024 at 11:35 PM Amit Kapila <[email protected]> wrote:
> > >
> >
> > > Also, adding wait sounds
> > > more like a boolean. So, I don't see the proposed names any better
> > > than the current one.
> > >
> >
> > Anyway, the point is that the current GUC name 'standby_slot_names' is
> > not ideal IMO because it doesn't have enough meaning by itself -- e.g.
> > you have to read the accompanying comment or documentation to have any
> > idea of its purpose.
> >
>
> Yeah, one has to read the description but that is true for other
> parameters like "temp_tablespaces". I don't have any better ideas but
> open to suggestions.
What about "non_lagging_standby_slots"?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-04 09:51 Amit Kapila <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Amit Kapila @ 2024-03-04 09:51 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Mon, Mar 4, 2024 at 9:35 AM Peter Smith <[email protected]> wrote:
>
> OK, if the code will remain as-is wouldn't it be better to anyway
> change the function name to indicate what it really does?
>
> e.g. NeedToWaitForWal --> NeedToWaitForWalFlushOrStandbys
>
This seems too long. I would prefer the current name NeedToWaitForWal
as waiting for WAL means waiting to flush the WAL and waiting to
replicate it to standby. On similar lines, the variable name
standby_slot_oldest_flush_lsn looks too long. How about
ss_oldest_flush_lsn (where ss indicates standy_slots)?
Apart from this, I have made minor modifications in the attached.
--
With Regards,
Amit Kapila.
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index aa477015bd..1530e7720c 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -2390,8 +2390,8 @@ validate_standby_slots(char **newval)
{
/*
* We cannot validate the replication slot if the replication slots'
- * data has not been initialized. This is ok as we will validate the
- * specified slot when waiting for them to catch up. See
+ * data has not been initialized. This is ok as we will anyway validate
+ * the specified slot when waiting for them to catch up. See
* StandbySlotsHaveCaughtup for details.
*/
}
@@ -2473,8 +2473,7 @@ assign_standby_slot_names(const char *newval, void *extra)
char *standby_slot_names_cpy = extra;
/*
- * The standby slots may have changed, so we need to recompute the oldest
- * LSN.
+ * The standby slots may have changed, so we must recompute the oldest LSN.
*/
standby_slot_oldest_flush_lsn = InvalidXLogRecPtr;
@@ -2664,6 +2663,7 @@ StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel)
if (caught_up_slot_num != list_length(standby_slot_names_list))
return false;
+ /* The standby_slot_oldest_flush_lsn must not retreat. */
Assert(XLogRecPtrIsInvalid(standby_slot_oldest_flush_lsn) ||
min_restart_lsn >= standby_slot_oldest_flush_lsn);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index b71408d533..580f9dabd3 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1805,10 +1805,10 @@ NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
/*
* Check if the standby slots have caught up to the flushed position. It
- * is good to wait up to flushed position and then let it send the changes
- * to logical subscribers one by one which are already covered in flushed
- * position without needing to wait on every change for standby
- * confirmation.
+ * is good to wait up to the flushed position and then let the WalSender
+ * send the changes to logical subscribers one by one which are already
+ * covered by the flushed position without needing to wait on every change
+ * for standby confirmation.
*/
if (NeedToWaitForStandby(target_lsn, flushed_lsn, wait_event))
return true;
Attachments:
[text/plain] v104-0001_Amit_1.patch.txt (2.3K, ../../CAA4eK1K3+t0zF=d3YMKMP6KRM4byubhuMPXGCmp1Xufhvsyz6Q@mail.gmail.com/2-v104-0001_Amit_1.patch.txt)
download | inline diff:
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index aa477015bd..1530e7720c 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -2390,8 +2390,8 @@ validate_standby_slots(char **newval)
{
/*
* We cannot validate the replication slot if the replication slots'
- * data has not been initialized. This is ok as we will validate the
- * specified slot when waiting for them to catch up. See
+ * data has not been initialized. This is ok as we will anyway validate
+ * the specified slot when waiting for them to catch up. See
* StandbySlotsHaveCaughtup for details.
*/
}
@@ -2473,8 +2473,7 @@ assign_standby_slot_names(const char *newval, void *extra)
char *standby_slot_names_cpy = extra;
/*
- * The standby slots may have changed, so we need to recompute the oldest
- * LSN.
+ * The standby slots may have changed, so we must recompute the oldest LSN.
*/
standby_slot_oldest_flush_lsn = InvalidXLogRecPtr;
@@ -2664,6 +2663,7 @@ StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel)
if (caught_up_slot_num != list_length(standby_slot_names_list))
return false;
+ /* The standby_slot_oldest_flush_lsn must not retreat. */
Assert(XLogRecPtrIsInvalid(standby_slot_oldest_flush_lsn) ||
min_restart_lsn >= standby_slot_oldest_flush_lsn);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index b71408d533..580f9dabd3 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1805,10 +1805,10 @@ NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
/*
* Check if the standby slots have caught up to the flushed position. It
- * is good to wait up to flushed position and then let it send the changes
- * to logical subscribers one by one which are already covered in flushed
- * position without needing to wait on every change for standby
- * confirmation.
+ * is good to wait up to the flushed position and then let the WalSender
+ * send the changes to logical subscribers one by one which are already
+ * covered by the flushed position without needing to wait on every change
+ * for standby confirmation.
*/
if (NeedToWaitForStandby(target_lsn, flushed_lsn, wait_event))
return true;
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-04 11:22 Bertrand Drouvot <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Bertrand Drouvot @ 2024-03-04 11:22 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; Peter Smith <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
Hi,
On Sun, Mar 03, 2024 at 07:56:32AM +0000, Zhijie Hou (Fujitsu) wrote:
> Here is the V104 patch which addressed above and Peter's comments.
Thanks!
A few more random comments:
1 ===
+ The function may be blocked if the specified slot is a failover enabled
s/blocked/waiting/ ?
2 ===
+ * specified slot when waiting for them to catch up. See
+ * StandbySlotsHaveCaughtup for details.
s/StandbySlotsHaveCaughtup/StandbySlotsHaveCaughtup()/ ?
3 ===
+ /* Now verify if the specified slots really exist and have correct type */
remove "really"?
4 ===
+ /*
+ * Don't need to wait for the standbys to catch up if there is no value in
+ * standby_slot_names.
+ */
+ if (standby_slot_names_list == NIL)
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if we are on a standby
+ * server, since we do not support syncing slots to cascading standbys.
+ */
+ if (RecoveryInProgress())
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if they are already
+ * beyond the specified WAL location.
+ */
+ if (!XLogRecPtrIsInvalid(standby_slot_oldest_flush_lsn) &&
+ standby_slot_oldest_flush_lsn >= wait_for_lsn)
+ return true;
What about using OR conditions instead?
5 ===
+static bool
+NeedToWaitForStandby(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
Not a big deal but does it need to return a bool? (I mean it all depends of
the *wait_event value). Is it for better code readability in the caller?
6 ===
+static bool
+NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
Same questions as for NeedToWaitForStandby().
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-04 13:26 Zhijie Hou (Fujitsu) <[email protected]>
parent: Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-04 13:26 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Amit Kapila <[email protected]>; Peter Smith <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Monday, March 4, 2024 7:22 PM Bertrand Drouvot <[email protected]> wrote:
>
> Hi,
>
> On Sun, Mar 03, 2024 at 07:56:32AM +0000, Zhijie Hou (Fujitsu) wrote:
> > Here is the V104 patch which addressed above and Peter's comments.
>
> Thanks!
>
> A few more random comments:
Thanks for the comments!
>
> 1 ===
>
> + The function may be blocked if the specified slot is a failover
> + enabled
>
> s/blocked/waiting/ ?
Changed.
>
> 2 ===
>
> + * specified slot when waiting for them to catch up. See
> + * StandbySlotsHaveCaughtup for details.
>
> s/StandbySlotsHaveCaughtup/StandbySlotsHaveCaughtup()/ ?
Changed.
>
> 3 ===
>
> + /* Now verify if the specified slots really exist and have
> + correct type */
>
> remove "really"?
Changed.
>
> 4 ===
>
> + /*
> + * Don't need to wait for the standbys to catch up if there is no value in
> + * standby_slot_names.
> + */
> + if (standby_slot_names_list == NIL)
> + return true;
> +
> + /*
> + * Don't need to wait for the standbys to catch up if we are on a
> standby
> + * server, since we do not support syncing slots to cascading standbys.
> + */
> + if (RecoveryInProgress())
> + return true;
> +
> + /*
> + * Don't need to wait for the standbys to catch up if they are already
> + * beyond the specified WAL location.
> + */
> + if (!XLogRecPtrIsInvalid(standby_slot_oldest_flush_lsn) &&
> + standby_slot_oldest_flush_lsn >= wait_for_lsn)
> + return true;
>
> What about using OR conditions instead?
>
> 5 ===
>
> +static bool
> +NeedToWaitForStandby(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
> + uint32 *wait_event)
>
> Not a big deal but does it need to return a bool? (I mean it all depends of the
> *wait_event value). Is it for better code readability in the caller?
>
> 6 ===
>
> +static bool
> +NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
> + uint32 *wait_event)
>
> Same questions as for NeedToWaitForStandby().
I also feel the current style looks a bit cleaner, so didn’t change these.
Best Regards,
Hou zj
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-04 13:26 Zhijie Hou (Fujitsu) <[email protected]>
parent: Peter Smith <[email protected]>
1 sibling, 0 replies; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-04 13:26 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Monday, March 4, 2024 9:55 AM Peter Smith <[email protected]> wrote:
>
> On Sun, Mar 3, 2024 at 6:51 PM Zhijie Hou (Fujitsu) <[email protected]>
> wrote:
> >
> > On Sunday, March 3, 2024 7:47 AM Peter Smith <[email protected]>
> wrote:
> > >
>
> > > 3.
> > > + <note>
> > > + <para>
> > > + Value <literal>*</literal> is not accepted as it is inappropriate to
> > > + block logical replication for physical slots that either lack
> > > + associated standbys or have standbys associated that are
> > > + not
> > > enabled
> > > + for replication slot synchronization. (see
> > > + <xref
> > > linkend="logicaldecoding-replication-slots-synchronization"/>).
> > > + </para>
> > > + </note>
> > >
> > > Why does the document need to provide an excuse/reason for the rule?
> > > You could just say something like:
> > >
> > > SUGGESTION
> > > The slots must be named explicitly. For example, specifying wildcard
> > > values like <literal>*</literal> is not permitted.
> >
> > As suggested by Amit, I moved this to code comments.
>
> Was the total removal of this note deliberate? I only suggested removing the
> *reason* for the rule, not the entire rule. Otherwise, the user won't know to
> avoid doing this until they try it and get an error.
OK, Added.
>
>
> > >
> > > 9. NeedToWaitForWal
> > >
> > > + /*
> > > + * Check if the standby slots have caught up to the flushed position.
> > > + It
> > > + * is good to wait up to flushed position and then let it send the
> > > + changes
> > > + * to logical subscribers one by one which are already covered in
> > > + flushed
> > > + * position without needing to wait on every change for standby
> > > + * confirmation. Note that after receiving the shutdown signal, an
> > > + ERROR
> > > + * is reported if any slots are dropped, invalidated, or inactive.
> > > + This
> > > + * measure is taken to prevent the walsender from waiting indefinitely.
> > > + */
> > > + if (NeedToWaitForStandby(target_lsn, flushed_lsn, wait_event))
> > > + return true;
> > >
> > > I felt it was confusing things for this function to also call to the
> > > other one -- it seems an overlapping/muddling of the purpose of these.
> > > I think it will be easier to understand if the calling code just
> > > calls to one or both of these functions as required.
> >
> > Same as Amit, I didn't change this.
>
> AFAICT my previous review comment was misinterpreted. Please see [1] for
> more details.
>
> ~~~~
>
> Here are some more review comments for v104-00001
Thanks!
>
> ======
> Commit message
>
> 1.
> Additionally, The SQL functions pg_logical_slot_get_changes,
> pg_logical_slot_peek_changes and pg_replication_slot_advance are modified
> to wait for the replication slots specified in 'standby_slot_names' to catch up
> before returning.
>
> ~
>
> Maybe that should be expressed using similar wording as the docs...
>
> SUGGESTION
> Additionally, The SQL functions ... are modified. Now, when used with
> failover-enabled logical slots, these functions will block until all physical slots
> specified in 'standby_slot_names' have confirmed WAL receipt.
Changed.
>
> ======
> doc/src/sgml/config.sgml
>
> 2.
> + <function>pg_logical_slot_peek_changes</function></link>,
> + when used with failover enabled logical slots, will block until all
> + physical slots specified in
> <varname>standby_slot_names</varname> have
> + confirmed WAL receipt.
>
> /failover enabled logical slots/failover-enabled logical slots/
Changed. Note that for this comment and remaining comments,
I used the later version we agreed(logical failover slot).
>
> ======
> doc/src/sgml/func.sgml
>
> 3.
> + The function may be blocked if the specified slot is a failover enabled
> + slot and <link
> linkend="guc-standby-slot-names"><varname>standby_slot_names</varna
> me></link>
> + is configured.
> </para></entry>
>
> /a failover enabled slot//a failover-enabled slot/
Changed.
>
> ~~~
>
> 4.
> + slot may return to an earlier position. The function may be blocked if
> + the specified slot is a failover enabled slot and
> + <link
> linkend="guc-standby-slot-names"><varname>standby_slot_names</varna
> me></link>
> + is configured.
>
> /a failover enabled slot//a failover-enabled slot/
Changed.
>
> ======
> src/backend/replication/slot.c
>
> 5.
> +/*
> + * Wait for physical standbys to confirm receiving the given lsn.
> + *
> + * Used by logical decoding SQL functions that acquired failover enabled slot.
> + * It waits for physical standbys corresponding to the physical slots
> +specified
> + * in the standby_slot_names GUC.
> + */
> +void
> +WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
>
> /failover enabled slot/failover-enabled slot/
Changed.
>
> ~~~
>
> 6.
> + /*
> + * Don't need to wait for the standby to catch up if the current
> + acquired
> + * slot is not a failover enabled slot, or there is no value in
> + * standby_slot_names.
> + */
>
> /failover enabled slot/failover-enabled slot/
Changed.
>
> ======
> src/backend/replication/slotfuncs.c
>
> 7.
> +
> + /*
> + * Wake up logical walsenders holding failover enabled slots after
> + * updating the restart_lsn of the physical slot.
> + */
> + PhysicalWakeupLogicalWalSnd();
>
> /failover enabled slots/failover-enabled slots/
Changed.
>
> ======
> src/backend/replication/walsender.c
>
> 8.
> +/*
> + * Wake up the logical walsender processes with failover enabled slots
> +if the
> + * currently acquired physical slot is specified in standby_slot_names GUC.
> + */
> +void
> +PhysicalWakeupLogicalWalSnd(void)
>
> /failover enabled slots/failover-enabled slots/
Changed.
>
> 9.
> +/*
> + * Returns true if not all standbys have caught up to the flushed
> +position
> + * (flushed_lsn) when the current acquired slot is a failover enabled
> +logical
> + * slot and we are streaming; otherwise, returns false.
> + *
> + * If returning true, the function sets the appropriate wait event in
> + * wait_event; otherwise, wait_event is set to 0.
> + */
> +static bool
> +NeedToWaitForStandby(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
> + uint32 *wait_event)
>
> 9a.
> /failover enabled logical slot/failover-enabled logical slot/
Changed.
>
> ~
>
> 9b.
> Probably that function name should be plural.
>
> /NeedToWaitForStandby/NeedToWaitForStandbys/
>
> ~~~
>
> 10.
> +/*
> + * Returns true if we need to wait for WALs to be flushed to disk, or
> +if not
> + * all standbys have caught up to the flushed position (flushed_lsn)
> +when the
> + * current acquired slot is a failover enabled logical slot and we are
> + * streaming; otherwise, returns false.
> + *
> + * If returning true, the function sets the appropriate wait event in
> + * wait_event; otherwise, wait_event is set to 0.
> + */
> +static bool
> +NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
> + uint32 *wait_event)
>
> /failover enabled logical slot/failover-enabled logical slot/
Changed.
>
> ~~~
>
> 11. WalSndWaitForWal
>
> + /*
> + * Within the loop, we wait for the necessary WALs to be flushed to
> + * disk first, followed by waiting for standbys to catch up if there
> + * are enought WALs or upon receiving the shutdown signal. To avoid
> + * the scenario where standbys need to catch up to a newer WAL
> + * location in each iteration, we update our idea of the currently
> + * flushed position only if we are not waiting for standbys to catch
> + * up.
> + */
>
> typo
>
> /enought/enough/
Fixed.
Best Regards,
Hou zj
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-04 13:28 Zhijie Hou (Fujitsu) <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 2 replies; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-04 13:28 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; Peter Smith <[email protected]>; +Cc: shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Monday, March 4, 2024 5:52 PM Amit Kapila <[email protected]> wrote:
>
> On Mon, Mar 4, 2024 at 9:35 AM Peter Smith <[email protected]>
> wrote:
> >
> > OK, if the code will remain as-is wouldn't it be better to anyway
> > change the function name to indicate what it really does?
> >
> > e.g. NeedToWaitForWal --> NeedToWaitForWalFlushOrStandbys
> >
>
> This seems too long. I would prefer the current name NeedToWaitForWal as
> waiting for WAL means waiting to flush the WAL and waiting to replicate it to
> standby. On similar lines, the variable name standby_slot_oldest_flush_lsn looks
> too long. How about ss_oldest_flush_lsn (where ss indicates standy_slots)?
>
> Apart from this, I have made minor modifications in the attached.
Thanks, I have merged it.
Attach the V105 patch set which addressed Peter, Amit and Bertrand's comments.
This version also includes the following changes:
* We found a string matching issue for query_until() and fixed it.
* Removed one un-used parameter from NeedToWaitForStandbys.
* Disable the sub before testing the pg_logical_slot_get_changes in 040.pl, this is to prevent
This test from catching the warning from another walsender.
* Ran pgindent.
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v105-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch (48.3K, ../../OS0PR01MB5716881EDFAC8682E9219A4E94232@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v105-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch)
download | inline diff:
From 2244d50361be881bd5ae768efdae3a139ff26c94 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Mon, 4 Mar 2024 18:01:28 +0800
Subject: [PATCH v105 1/2] Allow logical walsenders to wait for the physical
standby
This patch introduces a mechanism to ensure that physical standby servers,
which are potential failover candidates, have received and flushed changes
before making them visible to subscribers. By doing so, it guarantees that
the promoted standby server is not lagging behind the subscribers when a
failover is necessary.
A new parameter named 'standby_slot_names' is introduced. The logical
walsender now guarantees that all local changes are sent and flushed to
the standby servers corresponding to the replication slots specified in
'standby_slot_names' before sending those changes to the subscriber.
Additionally, The SQL functions pg_logical_slot_get_changes,
pg_logical_slot_peek_changes and pg_replication_slot_advance are modified.
Now, when used with logical failover slots, these functions will
block until all physical slots specified in 'standby_slot_names' have
confirmed WAL receipt.
---
doc/src/sgml/config.sgml | 45 +++
doc/src/sgml/func.sgml | 14 +-
doc/src/sgml/logicaldecoding.sgml | 12 +
.../replication/logical/logicalfuncs.c | 12 +
src/backend/replication/logical/slotsync.c | 11 +
src/backend/replication/slot.c | 375 +++++++++++++++++-
src/backend/replication/slotfuncs.c | 12 +
src/backend/replication/walsender.c | 166 +++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 14 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/replication/slot.h | 5 +
src/include/replication/walsender.h | 1 +
src/include/replication/walsender_private.h | 7 +
src/include/utils/guc_hooks.h | 3 +
src/test/recovery/t/006_logical_decoding.pl | 3 +-
.../t/040_standby_failover_slots_sync.pl | 243 +++++++++++-
17 files changed, 899 insertions(+), 27 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index b38cbd714a..a467df8bfa 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4559,6 +4559,51 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Lists the streaming replication standby server slot names that logical
+ WAL sender processes will wait for. Logical WAL sender processes will
+ send decoded changes to plugins only after the specified replication
+ slots confirm receiving WAL. This guarantees that logical replication
+ slots with failover enabled do not consume changes until those changes
+ are received and flushed to corresponding physical standbys. If a
+ logical replication connection is meant to switch to a physical standby
+ after the standby is promoted, the physical replication slot for the
+ standby should be listed here. Note that logical replication will not
+ proceed if the slots specified in the
+ <varname>standby_slot_names</varname> do not exist or are invalidated.
+ Additionally, the replication management functions
+ <link linkend="pg-replication-slot-advance">
+ <function>pg_replication_slot_advance</function></link>,
+ <link linkend="pg-logical-slot-get-changes">
+ <function>pg_logical_slot_get_changes</function></link>, and
+ <link linkend="pg-logical-slot-peek-changes">
+ <function>pg_logical_slot_peek_changes</function></link>,
+ when used with logical failover slots, will block until all
+ physical slots specified in <varname>standby_slot_names</varname> have
+ confirmed WAL receipt.
+ </para>
+ <para>
+ The standbys corresponding to the physical replication slots in
+ <varname>standby_slot_names</varname> must configure
+ <literal>sync_replication_slots = true</literal> so they can receive
+ logical failover slots changes from the primary.
+ </para>
+ <note>
+ <para>
+ The slots must be named explicitly. For example, specifying wildcard
+ values like <literal>*</literal> is not permitted.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index e5fa82c161..77b6f40216 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -28150,7 +28150,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-get-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_get_changes</primary>
</indexterm>
@@ -28173,11 +28173,14 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
the specified value. Note, however, that the actual number of
rows returned may be larger, since this limit is only checked after
adding the rows produced when decoding each new transaction commit.
+ The function may be waiting if the specified slot is a logical failover
+ slot and <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ is configured.
</para></entry>
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-peek-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_peek_changes</primary>
</indexterm>
@@ -28232,7 +28235,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-replication-slot-advance" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_replication_slot_advance</primary>
</indexterm>
@@ -28248,7 +28251,10 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
the name of the slot and the actual position that it was advanced to.
The updated slot position information is written out at the next
checkpoint if any advancing is done. So in the event of a crash, the
- slot may return to an earlier position.
+ slot may return to an earlier position. The function may be waiting if
+ the specified slot is a logical failover slot and
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ is configured.
</para></entry>
</row>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 930c0fa8a6..5da3ac7f4c 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -384,6 +384,18 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU
must be enabled on the standby. It is also necessary to specify a valid
<literal>dbname</literal> in the
<link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>.
+ It's highly recommended that the said physical replication slot is named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ list on the primary, to prevent the subscriber from consuming changes
+ faster than the hot standby. Even when correctly configured, some latency
+ is expected when sending changes to logical subscribers due to the waiting
+ on slots named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>.
+ When <varname>standby_slot_names</varname> is utilized, the
+ primary server will not completely shut down until the corresponding
+ standbys, associated with the physical replication slots specified
+ in <varname>standby_slot_names</varname>, have confirmed
+ receiving the WAL up to the latest flushed position on the primary server.
</para>
<para>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index a1ff631e5e..b4dd5cce75 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -105,6 +105,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
MemoryContext per_query_ctx;
MemoryContext oldcontext;
XLogRecPtr end_of_wal;
+ XLogRecPtr wait_for_wal_lsn;
LogicalDecodingContext *ctx;
ResourceOwner old_resowner = CurrentResourceOwner;
ArrayType *arr;
@@ -224,6 +225,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
NameStr(MyReplicationSlot->data.plugin),
format_procedure(fcinfo->flinfo->fn_oid))));
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to wait_for_wal_lsn.
+ */
+ if (XLogRecPtrIsInvalid(upto_lsn))
+ wait_for_wal_lsn = end_of_wal;
+ else
+ wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
+
+ WaitForStandbyConfirmation(wait_for_wal_lsn);
+
ctx->output_writer_private = p;
/*
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index ad0fc6a04b..5074c8409f 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -488,6 +488,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
latestFlushPtr = GetStandbyFlushRecPtr(NULL);
if (remote_slot->confirmed_lsn > latestFlushPtr)
{
+ /*
+ * Can get here only if GUC 'standby_slot_names' on the primary server
+ * was not configured correctly.
+ */
ereport(AmLogicalSlotSyncWorkerProcess() ? LOG : ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("skipping slot synchronization as the received slot sync"
@@ -857,6 +861,13 @@ validate_remote_info(WalReceiverConn *wrconn)
remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull));
Assert(!isnull);
+ /*
+ * Slot sync is currently not supported on a cascading standby. This is
+ * because if we allow it, the primary server needs to wait for all the
+ * cascading standbys, otherwise, logical subscribers can still be ahead
+ * of one of the cascading standbys which we plan to promote. Thus, to
+ * avoid this additional complexity, we restrict it for the time being.
+ */
if (remote_in_recovery)
ereport(ERROR,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 2614f98ddd..3483f99a84 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -46,13 +46,18 @@
#include "common/string.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/interrupt.h"
#include "replication/slotsync.h"
#include "replication/slot.h"
+#include "replication/walsender_private.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
+#include "utils/memutils.h"
+#include "utils/varlena.h"
/*
* Replication slot on-disk data structure.
@@ -115,10 +120,25 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
/* My backend's replication slot in the shared memory array */
ReplicationSlot *MyReplicationSlot = NULL;
-/* GUC variable */
+/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+/*
+ * This GUC lists streaming replication standby server slot names that
+ * logical WAL sender processes will wait for.
+ */
+char *standby_slot_names;
+
+/* This is parsed and cached list for raw standby_slot_names. */
+static List *standby_slot_names_list = NIL;
+
+/*
+ * Oldest LSN that has been confirmed to be flushed to the standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
+
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
@@ -2345,3 +2365,356 @@ GetSlotInvalidationCause(const char *conflict_reason)
Assert(found);
return result;
}
+
+/*
+ * A helper function to validate slots specified in GUC standby_slot_names.
+ */
+static bool
+validate_standby_slots(char **newval)
+{
+ char *rawname;
+ List *elemlist;
+ bool ok;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Verify syntax and parse string into a list of identifiers */
+ ok = SplitIdentifierString(rawname, ',', &elemlist);
+
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
+ else if (!ReplicationSlotCtl)
+ {
+ /*
+ * We cannot validate the replication slot if the replication slots'
+ * data has not been initialized. This is ok as we will anyway
+ * validate the specified slot when waiting for them to catch up. See
+ * StandbySlotsHaveCaughtup() for details.
+ */
+ }
+ else
+ {
+ /* Check that the specified slots exist and are logical slots */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, elemlist)
+ {
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ GUC_check_errdetail("replication slot \"%s\" does not exist",
+ name);
+ ok = false;
+ break;
+ }
+
+ if (!SlotIsPhysical(slot))
+ {
+ GUC_check_errdetail("\"%s\" is not a physical replication slot",
+ name);
+ ok = false;
+ break;
+ }
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+ }
+
+ pfree(rawname);
+ list_free(elemlist);
+ return ok;
+}
+
+/*
+ * GUC check_hook for standby_slot_names
+ */
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ if ((*newval)[0] == '\0')
+ return true;
+
+ /*
+ * "*" is not accepted as in that case the primary will not be able to
+ * know for which all standbys to wait for. It is inappropriate to block
+ * logical replication for physical slots that either lack associated
+ * standbys or have standbys associated that are not enabled for
+ * replication slot synchronization.
+ */
+ if (strcmp(*newval, "*") == 0)
+ {
+ GUC_check_errdetail("\"*\" is not accepted for standby_slot_names");
+ return false;
+ }
+
+ /* Now verify if the specified slots exist and have correct type */
+ if (!validate_standby_slots(newval))
+ return false;
+
+ *extra = guc_strdup(ERROR, *newval);
+
+ return true;
+}
+
+/*
+ * GUC assign_hook for standby_slot_names
+ */
+void
+assign_standby_slot_names(const char *newval, void *extra)
+{
+ List *standby_slots;
+ MemoryContext oldcxt;
+ char *standby_slot_names_cpy = extra;
+
+ /*
+ * The standby slots may have changed, so we must recompute the oldest
+ * LSN.
+ */
+ ss_oldest_flush_lsn = InvalidXLogRecPtr;
+
+ list_free(standby_slot_names_list);
+ standby_slot_names_list = NIL;
+
+ /* No value is specified for standby_slot_names. */
+ if (standby_slot_names_cpy == NULL)
+ return;
+
+ if (!SplitIdentifierString(standby_slot_names_cpy, ',', &standby_slots))
+ {
+ /* This should not happen if GUC checked check_standby_slot_names. */
+ elog(ERROR, "invalid list syntax");
+ }
+
+ /*
+ * Switch to the memory context under which GUC variables are allocated
+ * (GUCMemoryContext).
+ */
+ oldcxt = MemoryContextSwitchTo(GetMemoryChunkContext(standby_slot_names_cpy));
+ standby_slot_names_list = list_copy(standby_slots);
+ MemoryContextSwitchTo(oldcxt);
+}
+
+/*
+ * Return the standby_slot_names_list.
+ *
+ * Note that since we do not support syncing slots to cascading standbys, we
+ * return NIL if we are running in a standby to indicate that no standby slots
+ * need to be waited for.
+ */
+List *
+GetStandbySlotList(void)
+{
+ if (RecoveryInProgress())
+ return NIL;
+
+ return standby_slot_names_list;
+}
+
+/*
+ * Return true if the slots specified in standby_slot_names have caught up to
+ * the given WAL location, false otherwise.
+ *
+ * The elevel parameter specifies the error level used for logging messages
+ * related to slots that do not exist, are invalidated, or are inactive.
+ */
+bool
+StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel)
+{
+ int caught_up_slot_num = 0;
+ XLogRecPtr min_restart_lsn = InvalidXLogRecPtr;
+
+ /*
+ * Don't need to wait for the standbys to catch up if there is no value in
+ * standby_slot_names.
+ */
+ if (standby_slot_names_list == NIL)
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if we are on a standby
+ * server, since we do not support syncing slots to cascading standbys.
+ */
+ if (RecoveryInProgress())
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if they are already
+ * beyond the specified WAL location.
+ */
+ if (!XLogRecPtrIsInvalid(ss_oldest_flush_lsn) &&
+ ss_oldest_flush_lsn >= wait_for_lsn)
+ return true;
+
+ /*
+ * To prevent concurrent slot dropping and creation while filtering the
+ * slots, take the ReplicationSlotControlLock outside of the loop.
+ */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, standby_slot_names_list)
+ {
+ XLogRecPtr restart_lsn;
+ bool invalidated;
+ bool inactive;
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ /*
+ * If a slot name provided in standby_slot_names does not exist,
+ * report a message and exit the loop. A user can specify a slot
+ * name that does not exist just before the server startup. The
+ * GUC check_hook(validate_standby_slots) cannot validate such a
+ * slot during startup as the ReplicationSlotCtl shared memory is
+ * not initialized at that time. It is also possible for a user to
+ * drop the slot in standby_slot_names afterwards.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not exist",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider creating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (SlotIsLogical(slot))
+ {
+ /*
+ * If a logical slot name is provided in standby_slot_names,
+ * report a message and exit the loop. Similar to the non-existent
+ * case, a user can specify a logical slot name in
+ * standby_slot_names before the server startup, or drop an
+ * existing physical slot and recreate a logical slot with the
+ * same name.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot have logical replication slot \"%s\" in parameter %s",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting for correction on \"%s\".",
+ name),
+ errhint("Consider removing logical slot \"%s\" from parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ SpinLockAcquire(&slot->mutex);
+ restart_lsn = slot->data.restart_lsn;
+ invalidated = slot->data.invalidated != RS_INVAL_NONE;
+ inactive = slot->active_pid == 0;
+ SpinLockRelease(&slot->mutex);
+
+ if (invalidated)
+ {
+ /* Specified physical slot has been invalidated */
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("physical slot \"%s\" specified in parameter %s has been invalidated",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider dropping and recreating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (XLogRecPtrIsInvalid(restart_lsn) || restart_lsn < wait_for_lsn)
+ {
+ /* Log a message if no active_pid for this physical slot */
+ if (inactive)
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not have active_pid",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider starting standby associated with \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+
+ /* Continue if the current slot hasn't caught up. */
+ break;
+ }
+
+ Assert(restart_lsn >= wait_for_lsn);
+
+ if (XLogRecPtrIsInvalid(min_restart_lsn) ||
+ min_restart_lsn > restart_lsn)
+ min_restart_lsn = restart_lsn;
+
+ caught_up_slot_num++;
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /*
+ * Return false if not all the standbys have caught up to the specified
+ * WAL location.
+ */
+ if (caught_up_slot_num != list_length(standby_slot_names_list))
+ return false;
+
+ /* The ss_oldest_flush_lsn must not retreat. */
+ Assert(XLogRecPtrIsInvalid(ss_oldest_flush_lsn) ||
+ min_restart_lsn >= ss_oldest_flush_lsn);
+
+ ss_oldest_flush_lsn = min_restart_lsn;
+
+ return true;
+}
+
+/*
+ * Wait for physical standbys to confirm receiving the given lsn.
+ *
+ * Used by logical decoding SQL functions that acquired logical failover slot.
+ * It waits for physical standbys corresponding to the physical slots specified
+ * in the standby_slot_names GUC.
+ */
+void
+WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
+{
+ /*
+ * Don't need to wait for the standby to catch up if the current acquired
+ * slot is not a logical failover slot, or there is no value in
+ * standby_slot_names.
+ */
+ if (!MyReplicationSlot->data.failover || standby_slot_names_list == NIL)
+ return;
+
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+
+ for (;;)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /* Exit if done waiting for every slot. */
+ if (StandbySlotsHaveCaughtup(wait_for_lsn, WARNING))
+ break;
+
+ /*
+ * Wait for the slots in the standby_slot_names to catch up, but use a
+ * timeout (1s) so we can also check if the standby_slot_names has
+ * been changed.
+ */
+ ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000,
+ WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION);
+ }
+
+ ConditionVariableCancelSleep();
+}
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 768a304723..ad79e1fccd 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -464,6 +464,12 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* crash, but this makes the data consistent after a clean shutdown.
*/
ReplicationSlotMarkDirty();
+
+ /*
+ * Wake up logical walsenders holding logical failover slots after
+ * updating the restart_lsn of the physical slot.
+ */
+ PhysicalWakeupLogicalWalSnd();
}
return retlsn;
@@ -504,6 +510,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto)
.segment_close = wal_segment_close),
NULL, NULL, NULL);
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to moveto lsn.
+ */
+ WaitForStandbyConfirmation(moveto);
+
/*
* Start reading at the slot's restart_lsn, which we know to point to
* a valid record.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 0f1047179c..96f44ba85d 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1725,26 +1725,120 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
ProcessPendingWrites();
}
+/*
+ * Wake up the logical walsender processes with logical failover slots if the
+ * currently acquired physical slot is specified in standby_slot_names GUC.
+ */
+void
+PhysicalWakeupLogicalWalSnd(void)
+{
+ List *standby_slots;
+
+ Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot));
+
+ standby_slots = GetStandbySlotList();
+
+ foreach_ptr(char, name, standby_slots)
+ {
+ if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0)
+ {
+ ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv);
+ return;
+ }
+ }
+}
+
+/*
+ * Returns true if not all standbys have caught up to the flushed position
+ * (flushed_lsn) when the current acquired slot is a logical failover
+ * slot and we are streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForStandbys(XLogRecPtr flushed_lsn, uint32 *wait_event)
+{
+ int elevel = got_STOPPING ? ERROR : WARNING;
+ bool failover_slot;
+
+ failover_slot = (replication_active && MyReplicationSlot->data.failover);
+
+ /*
+ * Note that after receiving the shutdown signal, an ERROR is reported if
+ * any slots are dropped, invalidated, or inactive. This measure is taken
+ * to prevent the walsender from waiting indefinitely.
+ */
+ if (failover_slot && !StandbySlotsHaveCaughtup(flushed_lsn, elevel))
+ {
+ *wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
+ return true;
+ }
+
+ *wait_event = 0;
+ return false;
+}
+
+/*
+ * Returns true if we need to wait for WALs to be flushed to disk, or if not
+ * all standbys have caught up to the flushed position (flushed_lsn) when the
+ * current acquired slot is a logical failover slot and we are
+ * streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
+{
+ /* Check if we need to wait for WALs to be flushed to disk. */
+ if (target_lsn > flushed_lsn)
+ {
+ *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
+ return true;
+ }
+
+ /*
+ * Check if the standby slots have caught up to the flushed position. It
+ * is good to wait up to the flushed position and then let the WalSender
+ * send the changes to logical subscribers one by one which are already
+ * covered by the flushed position without needing to wait on every change
+ * for standby confirmation.
+ */
+ if (NeedToWaitForStandbys(flushed_lsn, wait_event))
+ return true;
+
+ *wait_event = 0;
+ return false;
+}
+
/*
* Wait till WAL < loc is flushed to disk so it can be safely sent to client.
*
- * Returns end LSN of flushed WAL. Normally this will be >= loc, but
- * if we detect a shutdown request (either from postmaster or client)
- * we will return early, so caller must always check.
+ * If the walsender holds a logical slot that has enabled failover, we also
+ * wait for all the specified streaming replication standby servers to
+ * confirm receipt of WAL up to RecentFlushPtr.
+ *
+ * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we
+ * detect a shutdown request (either from postmaster or client) we will return
+ * early, so caller must always check.
*/
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
int wakeEvents;
+ uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
- * have enough WAL available. This is particularly interesting if we're
- * far behind.
+ * have enough WAL available and all the standby servers have confirmed
+ * receipt of WAL up to RecentFlushPtr. This is particularly interesting
+ * if we're far behind.
*/
- if (RecentFlushPtr != InvalidXLogRecPtr &&
- loc <= RecentFlushPtr)
+ if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
return RecentFlushPtr;
/* Get a more recent flush pointer. */
@@ -1755,6 +1849,7 @@ WalSndWaitForWal(XLogRecPtr loc)
for (;;)
{
+ bool wait_for_standby_at_stop = false;
long sleeptime;
/* Clear any already-pending wakeups */
@@ -1781,21 +1876,37 @@ WalSndWaitForWal(XLogRecPtr loc)
if (got_STOPPING)
XLogBackgroundFlush();
- /* Update our idea of the currently flushed position. */
- if (!RecoveryInProgress())
- RecentFlushPtr = GetFlushRecPtr(NULL);
- else
- RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * Within the loop, we wait for the necessary WALs to be flushed to
+ * disk first, followed by waiting for standbys to catch up if there
+ * are enough WALs or upon receiving the shutdown signal. To avoid the
+ * scenario where standbys need to catch up to a newer WAL location in
+ * each iteration, we update our idea of the currently flushed
+ * position only if we are not waiting for standbys to catch up.
+ */
+ if (wait_event != WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ {
+ if (!RecoveryInProgress())
+ RecentFlushPtr = GetFlushRecPtr(NULL);
+ else
+ RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ }
/*
- * If postmaster asked us to stop, don't wait anymore.
+ * If postmaster asked us to stop and the standby slots have caught up
+ * to the flushed position, don't wait anymore.
*
* It's important to do this check after the recomputation of
* RecentFlushPtr, so we can send all remaining data before shutting
* down.
*/
if (got_STOPPING)
- break;
+ {
+ if (NeedToWaitForStandbys(RecentFlushPtr, &wait_event))
+ wait_for_standby_at_stop = true;
+ else
+ break;
+ }
/*
* We only send regular messages to the client for full decoded
@@ -1810,11 +1921,18 @@ WalSndWaitForWal(XLogRecPtr loc)
!waiting_for_ping_response)
WalSndKeepalive(false, InvalidXLogRecPtr);
- /* check whether we're done */
- if (loc <= RecentFlushPtr)
+ /*
+ * Exit the loop if already caught up and doesn't need to wait for
+ * standby slots.
+ */
+ if (!wait_for_standby_at_stop &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
break;
- /* Waiting for new WAL. Since we need to wait, we're now caught up. */
+ /*
+ * Waiting for new WAL or waiting for standbys to catch up. Since we
+ * need to wait, we're now caught up.
+ */
WalSndCaughtUp = true;
/*
@@ -1852,7 +1970,9 @@ WalSndWaitForWal(XLogRecPtr loc)
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL);
+ Assert(wait_event != 0);
+
+ WalSndWait(wakeEvents, sleeptime, wait_event);
}
/* reactivate latch so WalSndLoop knows to continue */
@@ -2262,6 +2382,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn)
{
ReplicationSlotMarkDirty();
ReplicationSlotsComputeRequiredLSN();
+ PhysicalWakeupLogicalWalSnd();
}
/*
@@ -3535,6 +3656,7 @@ WalSndShmemInit(void)
ConditionVariableInit(&WalSndCtl->wal_flush_cv);
ConditionVariableInit(&WalSndCtl->wal_replay_cv);
+ ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv);
}
}
@@ -3604,8 +3726,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event)
*
* And, we use separate shared memory CVs for physical and logical
* walsenders for selective wake ups, see WalSndWakeup() for more details.
+ *
+ * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV
+ * until awakened by physical walsenders after the walreceiver confirms
+ * the receipt of the LSN.
*/
- if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
+ if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+ else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv);
else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index ec2f31f82a..c08e00d1d6 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP
LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server."
LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server."
SSL_OPEN_SERVER "Waiting for SSL while attempting connection."
+WAIT_FOR_STANDBY_CONFIRMATION "Waiting for WAL to be received and flushed by the physical standby."
WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process."
WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 45013582a7..d77214795d 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4670,6 +4670,20 @@ struct config_string ConfigureNamesString[] =
check_debug_io_direct, assign_debug_io_direct, NULL
},
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("Lists streaming replication standby server slot "
+ "names that logical WAL sender processes will wait for."),
+ gettext_noop("Logical WAL sender processes will send decoded "
+ "changes to plugins only after the specified "
+ "replication slots confirm receiving WAL."),
+ GUC_LIST_INPUT
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, assign_standby_slot_names, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index edcc0282b2..2244ee52f7 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -343,6 +343,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsender processes will wait for
# - Standby Servers -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index acbf567150..61a6b97952 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -226,6 +226,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
+extern PGDLLIMPORT char *standby_slot_names;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -274,4 +275,8 @@ extern void CheckSlotPermissions(void);
extern ReplicationSlotInvalidationCause
GetSlotInvalidationCause(const char *conflict_reason);
+extern List *GetStandbySlotList(void);
+extern bool StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel);
+extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn);
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index 0c3996e926..f2d8297f01 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -39,6 +39,7 @@ extern void InitWalSender(void);
extern bool exec_replication_command(const char *cmd_string);
extern void WalSndErrorCleanup(void);
extern void WalSndResourceCleanup(bool isCommit);
+extern void PhysicalWakeupLogicalWalSnd(void);
extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
extern void WalSndSignals(void);
extern Size WalSndShmemSize(void);
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 3113e9ea47..109924ffcd 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -113,6 +113,13 @@ typedef struct
ConditionVariable wal_flush_cv;
ConditionVariable wal_replay_cv;
+ /*
+ * Used by physical walsenders holding slots specified in
+ * standby_slot_names to wake up logical walsenders holding logical
+ * failover slots when a walreceiver confirms the receipt of LSN.
+ */
+ ConditionVariable wal_confirm_rcv_cv;
+
WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER];
} WalSndCtlData;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index c8a7aa9a11..d64dc5fcdb 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra,
extern void assign_wal_consistency_checking(const char *newval, void *extra);
extern bool check_wal_segment_size(int *newval, void **extra, GucSource source);
extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
+extern bool check_standby_slot_names(char **newval, void **extra,
+ GucSource source);
+extern void assign_standby_slot_names(const char *newval, void *extra);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl
index 5c7b4ca5e3..b95d95c06f 100644
--- a/src/test/recovery/t/006_logical_decoding.pl
+++ b/src/test/recovery/t/006_logical_decoding.pl
@@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'},
undef, 'logical slot was actually dropped with DB');
# Test logical slot advancing and its durability.
+# Passing failover=true (last arg) should not have any impact on advancing.
my $logical_slot = 'logical_slot';
$node_primary->safe_psql('postgres',
- "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);"
+ "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);"
);
$node_primary->psql(
'postgres', "
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 021c58f621..1cab62c88c 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -504,18 +504,257 @@ ok( $standby1->poll_query_until(
"SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot' AND synced AND NOT temporary;"),
'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby');
+##################################################
+# Test that logical failover replication slots wait for the specified
+# physical replication slots to receive the changes first. It uses the
+# following set up:
+#
+# (physical standbys)
+# | ----> standby1 (primary_slot_name = sb1_slot)
+# | ----> standby2 (primary_slot_name = sb2_slot)
+# primary ----- |
+# (logical replication)
+# | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
+# | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
+#
+# standby_slot_names = 'sb1_slot'
+#
+# The setup is configured in such a way that the logical slot of subscriber1 is
+# enabled for failover, and thus the subscriber1 will wait for the physical
+# slot of standby1(sb1_slot) to catch up before receiving the decoded changes.
+##################################################
+
+$backup_name = 'backup3';
+
+$primary->psql('postgres',
+ q{SELECT pg_create_physical_replication_slot('sb2_slot');});
+
+$primary->backup($backup_name);
+
+# Create another standby
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup(
+ $primary, $backup_name,
+ has_streaming => 1,
+ has_restoring => 1);
+$standby2->append_conf(
+ 'postgresql.conf', qq(
+primary_slot_name = 'sb2_slot'
+));
+$standby2->start;
+$primary->wait_for_replay_catchup($standby2);
+
+# Configure primary to disallow any logical slots that have enabled failover
+# from getting ahead of the specified physical replication slot (sb1_slot).
+$primary->append_conf(
+ 'postgresql.conf', qq(
+standby_slot_names = 'sb1_slot'
+));
+$primary->reload;
+
+# Create another subscriber node without enabling failover, wait for sync to
+# complete
+my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$subscriber2->init;
+$subscriber2->start;
+$subscriber2->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE tab_int (a int PRIMARY KEY);
+ CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot);
+]);
+
+$subscriber2->wait_for_subscription_sync;
+
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+my $offset = -s $primary->logfile;
+
+# Stop the standby associated with the specified physical replication slot
+# (sb1_slot) so that the logical replication slot (lsub1_slot) won't receive
+# changes until the standby comes up.
+$standby1->stop;
+
+# Create some data on the primary
+my $primary_row_count = 20;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(11, $primary_row_count);");
+
+# Wait until the standby2 that's still running gets the data from the primary
+$primary->wait_for_replay_catchup($standby2);
+$result = $standby2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby2 gets data from primary");
+
+# Wait for regress_mysub2 to get the data from the primary. This subscription
+# was not enabled for failover so it gets the data without waiting for any
+# standbys.
+$primary->wait_for_catchup('regress_mysub2');
+$result = $subscriber2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "subscriber2 gets data from primary");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# The regress_mysub1 was enabled for failover so it doesn't get the data from
+# primary and keeps waiting for the standby specified in standby_slot_names
+# (sb1_slot aka standby1).
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) <> $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data from primary until standby1 acknowledges changes"
+);
+
+# Start the standby specified in standby_slot_names (sb1_slot aka standby1) and
+# wait for it to catch up with the primary.
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+$result = $standby1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby1 gets data from primary");
+
+# Now that the standby specified in standby_slot_names is up and running, the
+# primary can send the decoded changes to the subscription enabled for failover
+# (i.e. regress_mysub1). While the standby was down, regress_mysub1 didn't
+# receive any data from the primary. i.e. the primary didn't allow it to go
+# ahead of standby.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 acknowledges changes");
+
+##################################################
+# Verify that when using pg_logical_slot_get_changes to consume changes from a
+# logical slot with failover enabled, it will also wait for the slots specified
+# in standby_slot_names to catch up.
+##################################################
+
+# Stop the standby associated with the specified physical replication slot so
+# that the logical replication slot won't receive changes until the standby
+# slot's restart_lsn is advanced or the slot is removed from the
+# standby_slot_names list.
+$primary->safe_psql('postgres', "TRUNCATE tab_int;");
+$primary->wait_for_catchup('regress_mysub1');
+$standby1->stop;
+
+# Disable the regress_mysub1 to prevent the logical walsender from generating
+# more warnings.
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 DISABLE");
+
+# Wait for the replication slot to become inactive on the publisher
+$primary->poll_query_until(
+ 'postgres',
+ "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'",
+ 1);
+
+# Create a logical 'test_decoding' replication slot with failover enabled
+$primary->safe_psql('postgres',
+ "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);"
+);
+
+my $back_q = $primary->background_psql(
+ 'postgres',
+ on_error_stop => 0,
+ timeout => $PostgreSQL::Test::Utils::timeout_default);
+
+# Try and get changes from the logical slot with failover enabled.
+$offset = -s $primary->logfile;
+$back_q->query_until(
+ qr/logical_slot_get_changes/, q(
+ \echo logical_slot_get_changes
+ SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);
+));
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the function
+# pg_logical_slot_get_changes should now return, and the session can be
+# stopped.
+$back_q->quit;
+
+$primary->safe_psql('postgres',
+ "SELECT pg_drop_replication_slot('test_slot');"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
+# Enable the regress_mysub1 for further tests
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+##################################################
+# Test that logical replication will wait for the user-created inactive
+# physical slot to catch up until we remove the slot from standby_slot_names.
+##################################################
+
+$offset = -s $primary->logfile;
+
+# Create some data on the primary
+$primary_row_count = 10;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# The regress_mysub1 doesn't get the data from primary because the specified
+# standby slot (sb1_slot) in standby_slot_names is inactive.
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data as the sb1_slot doesn't catch up");
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the primary server should now
+# send the decoded changes to the subscription.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
##################################################
# Promote the standby1 to primary. Confirm that:
# a) the slot 'lsub1_slot' is retained on the new primary
# b) logical replication for regress_mysub1 is resumed successfully after failover
##################################################
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+
$standby1->promote;
# Update subscription with the new primary's connection info
my $standby1_conninfo = $standby1->connstr . ' dbname=postgres';
$subscriber1->safe_psql('postgres',
- "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo';
- ALTER SUBSCRIPTION regress_mysub1 ENABLE; ");
+ "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo';");
# Confirm the synced slot 'lsub1_slot' is retained on the new primary
is($standby1->safe_psql('postgres',
--
2.30.0.windows.2
[application/octet-stream] v105-0002-Document-the-steps-to-check-if-the-standby-is-r.patch (7.0K, ../../OS0PR01MB5716881EDFAC8682E9219A4E94232@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v105-0002-Document-the-steps-to-check-if-the-standby-is-r.patch)
download | inline diff:
From 2789401edf995a70b7c73c43632ffd802c4b50dc Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Fri, 19 Jan 2024 11:04:16 +0530
Subject: [PATCH v105 2/2] Document the steps to check if the standby is ready
for failover
---
doc/src/sgml/high-availability.sgml | 9 ++
doc/src/sgml/logical-replication.sgml | 136 ++++++++++++++++++++++++++
2 files changed, 145 insertions(+)
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 236c0af65f..36215aa68c 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -1487,6 +1487,15 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)'
Written administration procedures are advised.
</para>
+ <para>
+ If you have opted for synchronization of logical slots (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>),
+ then before switching to the standby server, it is recommended to check
+ if the logical slots synchronized on the standby server are ready
+ for failover. This can be done by following the steps described in
+ <xref linkend="logical-replication-failover"/>.
+ </para>
+
<para>
To trigger failover of a log-shipping standby server, run
<command>pg_ctl promote</command> or call <function>pg_promote()</function>.
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ec2130669e..be59d306a1 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -687,6 +687,142 @@ ALTER SUBSCRIPTION
</sect1>
+ <sect1 id="logical-replication-failover">
+ <title>Logical Replication Failover</title>
+
+ <para>
+ When the publisher server is the primary server of a streaming replication,
+ the logical slots on that primary server can be synchronized to the standby
+ server by specifying <literal>failover = true</literal> when creating
+ subscriptions for those publications. Enabling failover ensures a seamless
+ transition of those subscriptions after the standby is promoted. They can
+ continue subscribing to publications now on the new primary server without
+ any data loss.
+ </para>
+
+ <para>
+ Because the slot synchronization logic copies asynchronously, it is
+ necessary to confirm that replication slots have been synced to the standby
+ server before the failover happens. Furthermore, to ensure a successful
+ failover, the standby server must not be lagging behind the subscriber. It
+ is highly recommended to use
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ to prevent the subscriber from consuming changes faster than the hot standby.
+ To confirm that the standby server is indeed ready for failover, follow
+ these 2 steps:
+ </para>
+
+ <procedure>
+ <step performance="required">
+ <para>
+ Confirm that all the necessary logical replication slots have been synced to
+ the standby server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node, use the following SQL to identify
+ which slots should be synced to the standby that we plan to promote.
+<programlisting>
+test_sub=# SELECT
+ array_agg(slotname) AS slots
+ FROM
+ ((
+ SELECT r.srsubid AS subid, CONCAT('pg_', srsubid, '_sync_', srrelid, '_', ctl.system_identifier) AS slotname
+ FROM pg_control_system() ctl, pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate = 'f' AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT s.oid AS subid, s.subslotname as slotname
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ slots
+-------
+ {sub1,sub2,sub3}
+(1 row)
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, check that the logical replication slots identified above exist on
+ the standby server and are ready for failover.
+<programlisting>
+test_standby=# SELECT slot_name, (synced AND NOT temporary AND conflict_reason IS NULL) AS failover_ready
+ FROM pg_replication_slots
+ WHERE slot_name IN ('sub1','sub2','sub3');
+ slot_name | failover_ready
+-------------+----------------
+ sub1 | t
+ sub2 | t
+ sub3 | t
+(3 rows)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+
+ <step performance="required">
+ <para>
+ Confirm that the standby server is not lagging behind the subscribers.
+ This step can be skipped if
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ has been correctly configured. If standby_slot_names is not configured
+ correctly, it is highly recommended to run this step after the primary
+ server is down, otherwise the results of the query may vary at different
+ points of time due to the ongoing replication on the logical subscribers
+ from the primary server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node check the last replayed WAL.
+ This step needs to be run on the database(s) that includes the failover
+ enabled subscription(s), to find the last replayed WAL on each database.
+<programlisting>
+test_sub=# SELECT
+ MAX(remote_lsn) AS remote_lsn_on_subscriber
+ FROM
+ ((
+ SELECT (CASE WHEN r.srsubstate = 'f' THEN pg_replication_origin_progress(CONCAT('pg_', r.srsubid, '_', r.srrelid), false)
+ WHEN r.srsubstate IN ('s', 'r') THEN r.srsublsn END) AS remote_lsn
+ FROM pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate IN ('f', 's', 'r') AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT pg_replication_origin_progress(CONCAT('pg_', s.oid), false) AS remote_lsn
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ remote_lsn_on_subscriber
+--------------------------
+ 0/3000388
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, on the standby server check that the last-received WAL location
+ is ahead of the replayed WAL location(s) on the subscriber identified
+ above. If the above SQL result was NULL, it means the subscriber has not
+ yet replayed any WAL, so the standby server must be ahead of the
+ subscriber, and this step can be skipped.
+<programlisting>
+test_standby=# SELECT pg_last_wal_receive_lsn() >= '0/3000388'::pg_lsn AS failover_ready;
+ failover_ready
+----------------
+ t
+(1 row)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+ </procedure>
+
+ <para>
+ If the result (<literal>failover_ready</literal>) of both above steps is
+ true, existing subscriptions will be able to continue without data loss.
+ </para>
+
+ </sect1>
+
<sect1 id="logical-replication-row-filter">
<title>Row Filters</title>
--
2.30.0.windows.2
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-04 15:43 Bertrand Drouvot <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
1 sibling, 1 reply; 68+ messages in thread
From: Bertrand Drouvot @ 2024-03-04 15:43 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; Peter Smith <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
Hi,
On Mon, Mar 04, 2024 at 01:28:04PM +0000, Zhijie Hou (Fujitsu) wrote:
> Attach the V105 patch set
Thanks!
Sorry I missed those during the previous review:
1 ===
Commit message: "these functions will block until"
s/block/wait/ ?
2 ===
+ when used with logical failover slots, will block until all
s/block/wait/ ?
It seems those are the 2 remaining "block" that could deserve the proposed
above change.
3 ===
+ invalidated = slot->data.invalidated != RS_INVAL_NONE;
+ inactive = slot->active_pid == 0;
invalidated = (slot->data.invalidated != RS_INVAL_NONE);
inactive = (slot->active_pid == 0);
instead?
I think it's easier to read and it looks like this is the way it's written in
other places (at least the few I checked).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-05 00:40 Peter Smith <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
1 sibling, 1 reply; 68+ messages in thread
From: Peter Smith @ 2024-03-05 00:40 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
Here are some review comments for v105-0001
==========
doc/src/sgml/config.sgml
1.
+ <para>
+ The standbys corresponding to the physical replication slots in
+ <varname>standby_slot_names</varname> must configure
+ <literal>sync_replication_slots = true</literal> so they can receive
+ logical failover slots changes from the primary.
+ </para>
/slots changes/slot changes/
======
doc/src/sgml/func.sgml
2.
+ The function may be waiting if the specified slot is a logical failover
+ slot and <link
linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ is configured.
I know this has been through multiple versions already, but this
latest wording "may be waiting..." doesn't seem very good to me.
How about one of these?
* The function may not be able to return immediately if the specified
slot is a logical failover slot and standby_slot_names is configured.
* The function return might be blocked if the specified slot is a
logical failover slot and standby_slot_names is configured.
* If the specified slot is a logical failover slot then the function
will block until all physical slots specified in standby_slot_names
have confirmed WAL receipt.
* If the specified slot is a logical failover slot then the function
will not return until all physical slots specified in
standby_slot_names have confirmed WAL receipt.
~~~
3.
+ slot may return to an earlier position. The function may be waiting if
+ the specified slot is a logical failover slot and
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
Same as previous review comment #2
======
src/backend/replication/slot.c
4. WaitForStandbyConfirmation
+ * Used by logical decoding SQL functions that acquired logical failover slot.
IIUC it doesn't work like that. pg_logical_slot_get_changes_guts()
calls here unconditionally (i.e. the SQL functions don't even check if
they are failover slots before calling this) so the comment seems
misleading/redundant.
======
src/backend/replication/walsender.c
5. NeedToWaitForWal
+ /*
+ * Check if the standby slots have caught up to the flushed position. It
+ * is good to wait up to the flushed position and then let the WalSender
+ * send the changes to logical subscribers one by one which are already
+ * covered by the flushed position without needing to wait on every change
+ * for standby confirmation.
+ */
+ if (NeedToWaitForStandbys(flushed_lsn, wait_event))
+ return true;
+
+ *wait_event = 0;
+ return false;
+}
+
5a.
The comment (or part of it?) seems misplaced because it is talking
WalSender sending changes, but that is not happening in this function.
Also, isn't what this is saying already described by the other comment
in the caller? e.g.:
+ /*
+ * Within the loop, we wait for the necessary WALs to be flushed to
+ * disk first, followed by waiting for standbys to catch up if there
+ * are enough WALs or upon receiving the shutdown signal. To avoid the
+ * scenario where standbys need to catch up to a newer WAL location in
+ * each iteration, we update our idea of the currently flushed
+ * position only if we are not waiting for standbys to catch up.
+ */
~
5b.
Most of the code is unnecessary. AFAICT all this is exactly same as just 1 line:
return NeedToWaitForStandbys(flushed_lsn, wait_event);
~~~
6. WalSndWaitForWal
+ /*
+ * Within the loop, we wait for the necessary WALs to be flushed to
+ * disk first, followed by waiting for standbys to catch up if there
+ * are enough WALs or upon receiving the shutdown signal. To avoid the
+ * scenario where standbys need to catch up to a newer WAL location in
+ * each iteration, we update our idea of the currently flushed
+ * position only if we are not waiting for standbys to catch up.
+ */
Regarding that 1st sentence: maybe this logic used to be done
explicitly "within the loop" but IIUC this logic is now hidden inside
NeedToWaitForWal() so the comment should mention that.
----------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-05 03:24 Amit Kapila <[email protected]>
parent: Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 68+ messages in thread
From: Amit Kapila @ 2024-03-05 03:24 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Peter Smith <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Mon, Mar 4, 2024 at 2:27 PM Bertrand Drouvot
<[email protected]> wrote:
>
> On Thu, Feb 29, 2024 at 03:38:59PM +0530, Amit Kapila wrote:
> > On Thu, Feb 29, 2024 at 9:13 AM Peter Smith <[email protected]> wrote:
> > >
> > > On Tue, Feb 27, 2024 at 11:35 PM Amit Kapila <[email protected]> wrote:
> > > >
> > >
> > > > Also, adding wait sounds
> > > > more like a boolean. So, I don't see the proposed names any better
> > > > than the current one.
> > > >
> > >
> > > Anyway, the point is that the current GUC name 'standby_slot_names' is
> > > not ideal IMO because it doesn't have enough meaning by itself -- e.g.
> > > you have to read the accompanying comment or documentation to have any
> > > idea of its purpose.
> > >
> >
> > Yeah, one has to read the description but that is true for other
> > parameters like "temp_tablespaces". I don't have any better ideas but
> > open to suggestions.
>
> What about "non_lagging_standby_slots"?
>
I still prefer the current one as that at least resembles with
existing synchronous_standby_names. I think we can change the GUC name
if we get an agreement on a better name before release. At this stage,
let's move with the current one.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-05 03:45 Amit Kapila <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Amit Kapila @ 2024-03-05 03:45 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Tue, Mar 5, 2024 at 6:10 AM Peter Smith <[email protected]> wrote:
>
> ======
> src/backend/replication/walsender.c
>
> 5. NeedToWaitForWal
>
> + /*
> + * Check if the standby slots have caught up to the flushed position. It
> + * is good to wait up to the flushed position and then let the WalSender
> + * send the changes to logical subscribers one by one which are already
> + * covered by the flushed position without needing to wait on every change
> + * for standby confirmation.
> + */
> + if (NeedToWaitForStandbys(flushed_lsn, wait_event))
> + return true;
> +
> + *wait_event = 0;
> + return false;
> +}
> +
>
> 5a.
> The comment (or part of it?) seems misplaced because it is talking
> WalSender sending changes, but that is not happening in this function.
>
I don't think so. This is invoked only by walsender and a static
function. I don't see any other better place to mention this.
> Also, isn't what this is saying already described by the other comment
> in the caller? e.g.:
>
Oh no, here we are explaining the wait order.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-05 06:35 shveta malik <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 2 replies; 68+ messages in thread
From: shveta malik @ 2024-03-05 06:35 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Peter Smith <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>; shveta malik <[email protected]>
On Tue, Mar 5, 2024 at 9:15 AM Amit Kapila <[email protected]> wrote:
>
> On Tue, Mar 5, 2024 at 6:10 AM Peter Smith <[email protected]> wrote:
> >
> > ======
> > src/backend/replication/walsender.c
> >
> > 5. NeedToWaitForWal
> >
> > + /*
> > + * Check if the standby slots have caught up to the flushed position. It
> > + * is good to wait up to the flushed position and then let the WalSender
> > + * send the changes to logical subscribers one by one which are already
> > + * covered by the flushed position without needing to wait on every change
> > + * for standby confirmation.
> > + */
> > + if (NeedToWaitForStandbys(flushed_lsn, wait_event))
> > + return true;
> > +
> > + *wait_event = 0;
> > + return false;
> > +}
> > +
> >
> > 5a.
> > The comment (or part of it?) seems misplaced because it is talking
> > WalSender sending changes, but that is not happening in this function.
> >
>
> I don't think so. This is invoked only by walsender and a static
> function. I don't see any other better place to mention this.
>
> > Also, isn't what this is saying already described by the other comment
> > in the caller? e.g.:
> >
>
> Oh no, here we are explaining the wait order.
I think there is a scope of improvement here. The comment inside
NeedToWaitForWal() which states that we need to wait here for standbys
on flush-position(and not on each change) should be outside of this
function. It is too embedded. And the comment which states the order
of wait (first flush and then standbys confirmation) should be outside
the for-loop in WalSndWaitForWal(), but yes we do need both the
comments. Attached a patch (.txt) for comments improvement, please
merge if appropriate.
thanks
Shveta
From e03332839dd804f3bc38937e677ba87ac10f981b Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Tue, 5 Mar 2024 11:29:52 +0530
Subject: [PATCH v2] Comments improvement.
---
src/backend/replication/walsender.c | 28 ++++++++++++++--------------
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 96f44ba85d..6a082b42eb 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1799,13 +1799,6 @@ NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
return true;
}
- /*
- * Check if the standby slots have caught up to the flushed position. It
- * is good to wait up to the flushed position and then let the WalSender
- * send the changes to logical subscribers one by one which are already
- * covered by the flushed position without needing to wait on every change
- * for standby confirmation.
- */
if (NeedToWaitForStandbys(flushed_lsn, wait_event))
return true;
@@ -1818,7 +1811,10 @@ NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
*
* If the walsender holds a logical slot that has enabled failover, we also
* wait for all the specified streaming replication standby servers to
- * confirm receipt of WAL up to RecentFlushPtr.
+ * confirm receipt of WAL up to RecentFlushPtr. It's beneficial to wait
+ * here for the confirmation from standbys up to RecentFlushPtr rather
+ * than waiting in ReorderBufferCommit() before transmitting each
+ * change to logical subscribers, which is already covered in RecentFlushPtr.
*
* Returns end LSN of flushed WAL. Normally this will be >= loc, but if we
* detect a shutdown request (either from postmaster or client) we will return
@@ -1847,6 +1843,12 @@ WalSndWaitForWal(XLogRecPtr loc)
else
RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * In the loop, we wait for the necessary WALs to be flushed to disk
+ * initially, and then we wait for standbys to catch up. Upon receiving
+ * the shutdown signal, we immediately transition to waiting for standbys
+ * to catch up.
+ */
for (;;)
{
bool wait_for_standby_at_stop = false;
@@ -1877,12 +1879,10 @@ WalSndWaitForWal(XLogRecPtr loc)
XLogBackgroundFlush();
/*
- * Within the loop, we wait for the necessary WALs to be flushed to
- * disk first, followed by waiting for standbys to catch up if there
- * are enough WALs or upon receiving the shutdown signal. To avoid the
- * scenario where standbys need to catch up to a newer WAL location in
- * each iteration, we update our idea of the currently flushed
- * position only if we are not waiting for standbys to catch up.
+ * Update our idea of the currently flushed position, but do it only
+ * if we haven't started waiting for standbys. This is done to prevent
+ * standbys from needing to catch up to a more recent WAL location
+ * with each iteration.
*/
if (wait_event != WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
{
--
2.34.1
Attachments:
[text/plain] v2-0001-Comments-improvement.patch.txt (3.0K, ../../CAJpy0uDD3BjzJcbbQRzqN_3+pMRLGX08T+t87h-Or8rOO2c8hg@mail.gmail.com/2-v2-0001-Comments-improvement.patch.txt)
download | inline diff:
From e03332839dd804f3bc38937e677ba87ac10f981b Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Tue, 5 Mar 2024 11:29:52 +0530
Subject: [PATCH v2] Comments improvement.
---
src/backend/replication/walsender.c | 28 ++++++++++++++--------------
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 96f44ba85d..6a082b42eb 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1799,13 +1799,6 @@ NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
return true;
}
- /*
- * Check if the standby slots have caught up to the flushed position. It
- * is good to wait up to the flushed position and then let the WalSender
- * send the changes to logical subscribers one by one which are already
- * covered by the flushed position without needing to wait on every change
- * for standby confirmation.
- */
if (NeedToWaitForStandbys(flushed_lsn, wait_event))
return true;
@@ -1818,7 +1811,10 @@ NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
*
* If the walsender holds a logical slot that has enabled failover, we also
* wait for all the specified streaming replication standby servers to
- * confirm receipt of WAL up to RecentFlushPtr.
+ * confirm receipt of WAL up to RecentFlushPtr. It's beneficial to wait
+ * here for the confirmation from standbys up to RecentFlushPtr rather
+ * than waiting in ReorderBufferCommit() before transmitting each
+ * change to logical subscribers, which is already covered in RecentFlushPtr.
*
* Returns end LSN of flushed WAL. Normally this will be >= loc, but if we
* detect a shutdown request (either from postmaster or client) we will return
@@ -1847,6 +1843,12 @@ WalSndWaitForWal(XLogRecPtr loc)
else
RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * In the loop, we wait for the necessary WALs to be flushed to disk
+ * initially, and then we wait for standbys to catch up. Upon receiving
+ * the shutdown signal, we immediately transition to waiting for standbys
+ * to catch up.
+ */
for (;;)
{
bool wait_for_standby_at_stop = false;
@@ -1877,12 +1879,10 @@ WalSndWaitForWal(XLogRecPtr loc)
XLogBackgroundFlush();
/*
- * Within the loop, we wait for the necessary WALs to be flushed to
- * disk first, followed by waiting for standbys to catch up if there
- * are enough WALs or upon receiving the shutdown signal. To avoid the
- * scenario where standbys need to catch up to a newer WAL location in
- * each iteration, we update our idea of the currently flushed
- * position only if we are not waiting for standbys to catch up.
+ * Update our idea of the currently flushed position, but do it only
+ * if we haven't started waiting for standbys. This is done to prevent
+ * standbys from needing to catch up to a more recent WAL location
+ * with each iteration.
*/
if (wait_event != WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
{
--
2.34.1
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-05 06:45 Nisha Moond <[email protected]>
parent: shveta malik <[email protected]>
1 sibling, 0 replies; 68+ messages in thread
From: Nisha Moond @ 2024-03-05 06:45 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Amit Kapila <[email protected]>; Peter Smith <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
I did performance tests for the v99 patch w.r.t. wait time analysis.
As this patch is introducing a wait for standby before sending changes
to a subscriber, at the primary node, logged time at the start and end
of the XLogSendLogical() call (which eventually calls
WalSndWaitForWal()) and calculated total time taken by this function
during the load run.
For load, ran pgbench for 15 minutes:
Creating tables: pgbench -p 5833 postgres -qis 2
Running benchmark: pgbench postgres -p 5833 -c 10 -j 3 -T 900 -P 20
Machine details:
11th Gen Intel(R) Core(TM) i9-11950H @ 2.60GHz 32GB RAM
OS - Windows 10 Enterprise
Test setup:
Primary node -->
-> One physical standby node
-> One subscriber node having only one subscription with failover=true
-- the slot-sync relevant parameters are set to default (OFF) for all
the tests i.e.
hot_standby_feedback = off
sync_replication_slots = false
-- addition configuration on each instance is:
shared_buffers = 6GB
max_worker_processes = 32
max_parallel_maintenance_workers = 24
max_parallel_workers = 32
synchronous_commit = off
checkpoint_timeout = 1d
max_wal_size = 24GB
min_wal_size = 15GB
autovacuum = off
To review the wait time impact with and without patch, compared three
cases (did two runs for each case)-
(1) HEAD code:
time taken in run 1 = 103.935631 seconds
time taken in run 2 = 104.832186 seconds
(2) HEAD code + v99 patch ('standby_slot_names' is not set):
time taken in run 1 = 104.076343 seconds
time taken in run 2 = 103.116226 seconds
(3) HEAD code + v99 patch + a valid 'standby_slot_names' is set:
time taken in run 1 = 103.871012 seconds
time taken in run 2 = 103.793524 seconds
The time consumption of XLogSendLogical() is almost same in all the
cases and no performance degradation is observed.
--
Thanks,
Nisha
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-05 07:15 Zhijie Hou (Fujitsu) <[email protected]>
parent: Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-05 07:15 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Amit Kapila <[email protected]>; Peter Smith <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Monday, March 4, 2024 11:44 PM Bertrand Drouvot <[email protected]> wrote:
>
> Hi,
>
> On Mon, Mar 04, 2024 at 01:28:04PM +0000, Zhijie Hou (Fujitsu) wrote:
> > Attach the V105 patch set
>
> Thanks!
>
> Sorry I missed those during the previous review:
No problem, thanks for the comments!
>
> 1 ===
>
> Commit message: "these functions will block until"
>
> s/block/wait/ ?
>
> 2 ===
>
> + when used with logical failover slots, will block until all
>
> s/block/wait/ ?
>
> It seems those are the 2 remaining "block" that could deserve the proposed
> above change.
I prefer using 'block' here. And it seems others also suggest
to change the 'wait'[1].
>
> 3 ===
>
> + invalidated = slot->data.invalidated != RS_INVAL_NONE;
> + inactive = slot->active_pid == 0;
>
> invalidated = (slot->data.invalidated != RS_INVAL_NONE); inactive =
> (slot->active_pid == 0);
>
> instead?
>
> I think it's easier to read and it looks like this is the way it's written in other
> places (at least the few I checked).
I think the current code is consistent with other similar code in slot.c.
(grep "data.invalidated != RS_INVAL_NONE").
[1] https://www.postgresql.org/message-id/CAHut%2BPsATK8z1TEcfFE8zWoS1hagqsvaWYCgom_zYtScfwO7uQ%40mail.g...
Best Regards,
Hou zj
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-05 07:21 Zhijie Hou (Fujitsu) <[email protected]>
parent: shveta malik <[email protected]>
1 sibling, 0 replies; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-05 07:21 UTC (permalink / raw)
To: shveta malik <[email protected]>; Amit Kapila <[email protected]>; +Cc: Peter Smith <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Tuesday, March 5, 2024 2:35 PM shveta malik <[email protected]> wrote:
>
> On Tue, Mar 5, 2024 at 9:15 AM Amit Kapila <[email protected]> wrote:
> >
> > On Tue, Mar 5, 2024 at 6:10 AM Peter Smith <[email protected]>
> wrote:
> > >
> > > ======
> > > src/backend/replication/walsender.c
> > >
> > > 5. NeedToWaitForWal
> > >
> > > + /*
> > > + * Check if the standby slots have caught up to the flushed
> > > + position. It
> > > + * is good to wait up to the flushed position and then let the
> > > + WalSender
> > > + * send the changes to logical subscribers one by one which are
> > > + already
> > > + * covered by the flushed position without needing to wait on every
> > > + change
> > > + * for standby confirmation.
> > > + */
> > > + if (NeedToWaitForStandbys(flushed_lsn, wait_event)) return true;
> > > +
> > > + *wait_event = 0;
> > > + return false;
> > > +}
> > > +
> > >
> > > 5a.
> > > The comment (or part of it?) seems misplaced because it is talking
> > > WalSender sending changes, but that is not happening in this function.
> > >
> >
> > I don't think so. This is invoked only by walsender and a static
> > function. I don't see any other better place to mention this.
> >
> > > Also, isn't what this is saying already described by the other
> > > comment in the caller? e.g.:
> > >
> >
> > Oh no, here we are explaining the wait order.
>
> I think there is a scope of improvement here. The comment inside
> NeedToWaitForWal() which states that we need to wait here for standbys on
> flush-position(and not on each change) should be outside of this function. It is
> too embedded. And the comment which states the order of wait (first flush and
> then standbys confirmation) should be outside the for-loop in
> WalSndWaitForWal(), but yes we do need both the comments. Attached a
> patch (.txt) for comments improvement, please merge if appropriate.
Thanks, I have slightly modified the top-up patch and merged it.
Attach the V106 patch which addressed above and Peter's comments[1].
[1] https://www.postgresql.org/message-id/CAHut%2BPsATK8z1TEcfFE8zWoS1hagqsvaWYCgom_zYtScfwO7uQ%40mail.g...
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v106-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch (48.5K, ../../OS0PR01MB571613E2CA15D369D943EF4394222@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v106-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch)
download | inline diff:
From d3879e909250e487db2d2b26bcffefa32dcf1f8f Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Mon, 4 Mar 2024 18:01:28 +0800
Subject: [PATCH v106 1/2] Allow logical walsenders to wait for the physical
standby
This patch introduces a mechanism to ensure that physical standby servers,
which are potential failover candidates, have received and flushed changes
before making them visible to subscribers. By doing so, it guarantees that
the promoted standby server is not lagging behind the subscribers when a
failover is necessary.
A new parameter named 'standby_slot_names' is introduced. The logical
walsender now guarantees that all local changes are sent and flushed to
the standby servers corresponding to the replication slots specified in
'standby_slot_names' before sending those changes to the subscriber.
Additionally, The SQL functions pg_logical_slot_get_changes,
pg_logical_slot_peek_changes and pg_replication_slot_advance are modified.
Now, when used with logical failover slots, these functions will
block until all physical slots specified in 'standby_slot_names' have
confirmed WAL receipt.
---
doc/src/sgml/config.sgml | 45 +++
doc/src/sgml/func.sgml | 16 +-
doc/src/sgml/logicaldecoding.sgml | 12 +
.../replication/logical/logicalfuncs.c | 12 +
src/backend/replication/logical/slotsync.c | 11 +
src/backend/replication/slot.c | 374 +++++++++++++++++-
src/backend/replication/slotfuncs.c | 12 +
src/backend/replication/walsender.c | 162 +++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 14 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/replication/slot.h | 5 +
src/include/replication/walsender.h | 1 +
src/include/replication/walsender_private.h | 7 +
src/include/utils/guc_hooks.h | 3 +
src/test/recovery/t/006_logical_decoding.pl | 3 +-
.../t/040_standby_failover_slots_sync.pl | 244 +++++++++++-
17 files changed, 897 insertions(+), 27 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index b38cbd714a..792ab22290 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4559,6 +4559,51 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Lists the streaming replication standby server slot names that logical
+ WAL sender processes will wait for. Logical WAL sender processes will
+ send decoded changes to plugins only after the specified replication
+ slots confirm receiving WAL. This guarantees that logical replication
+ slots with failover enabled do not consume changes until those changes
+ are received and flushed to corresponding physical standbys. If a
+ logical replication connection is meant to switch to a physical standby
+ after the standby is promoted, the physical replication slot for the
+ standby should be listed here. Note that logical replication will not
+ proceed if the slots specified in the
+ <varname>standby_slot_names</varname> do not exist or are invalidated.
+ Additionally, the replication management functions
+ <link linkend="pg-replication-slot-advance">
+ <function>pg_replication_slot_advance</function></link>,
+ <link linkend="pg-logical-slot-get-changes">
+ <function>pg_logical_slot_get_changes</function></link>, and
+ <link linkend="pg-logical-slot-peek-changes">
+ <function>pg_logical_slot_peek_changes</function></link>,
+ when used with logical failover slots, will block until all
+ physical slots specified in <varname>standby_slot_names</varname> have
+ confirmed WAL receipt.
+ </para>
+ <para>
+ The standbys corresponding to the physical replication slots in
+ <varname>standby_slot_names</varname> must configure
+ <literal>sync_replication_slots = true</literal> so they can receive
+ logical failover slot changes from the primary.
+ </para>
+ <note>
+ <para>
+ The slots must be named explicitly. For example, specifying wildcard
+ values like <literal>*</literal> is not permitted.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index e5fa82c161..0bb7aeb40e 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -28150,7 +28150,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-get-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_get_changes</primary>
</indexterm>
@@ -28173,11 +28173,15 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
the specified value. Note, however, that the actual number of
rows returned may be larger, since this limit is only checked after
adding the rows produced when decoding each new transaction commit.
+ If the specified slot is a logical failover slot then the function will
+ not return until all physical slots specified in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ have confirmed WAL receipt.
</para></entry>
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-peek-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_peek_changes</primary>
</indexterm>
@@ -28232,7 +28236,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-replication-slot-advance" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_replication_slot_advance</primary>
</indexterm>
@@ -28248,7 +28252,11 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
the name of the slot and the actual position that it was advanced to.
The updated slot position information is written out at the next
checkpoint if any advancing is done. So in the event of a crash, the
- slot may return to an earlier position.
+ slot may return to an earlier position. If the specified slot is a
+ logical failover slot then the function will not return until all
+ physical slots specified in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ have confirmed WAL receipt.
</para></entry>
</row>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 930c0fa8a6..5da3ac7f4c 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -384,6 +384,18 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU
must be enabled on the standby. It is also necessary to specify a valid
<literal>dbname</literal> in the
<link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>.
+ It's highly recommended that the said physical replication slot is named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ list on the primary, to prevent the subscriber from consuming changes
+ faster than the hot standby. Even when correctly configured, some latency
+ is expected when sending changes to logical subscribers due to the waiting
+ on slots named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>.
+ When <varname>standby_slot_names</varname> is utilized, the
+ primary server will not completely shut down until the corresponding
+ standbys, associated with the physical replication slots specified
+ in <varname>standby_slot_names</varname>, have confirmed
+ receiving the WAL up to the latest flushed position on the primary server.
</para>
<para>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index a1ff631e5e..b4dd5cce75 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -105,6 +105,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
MemoryContext per_query_ctx;
MemoryContext oldcontext;
XLogRecPtr end_of_wal;
+ XLogRecPtr wait_for_wal_lsn;
LogicalDecodingContext *ctx;
ResourceOwner old_resowner = CurrentResourceOwner;
ArrayType *arr;
@@ -224,6 +225,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
NameStr(MyReplicationSlot->data.plugin),
format_procedure(fcinfo->flinfo->fn_oid))));
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to wait_for_wal_lsn.
+ */
+ if (XLogRecPtrIsInvalid(upto_lsn))
+ wait_for_wal_lsn = end_of_wal;
+ else
+ wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
+
+ WaitForStandbyConfirmation(wait_for_wal_lsn);
+
ctx->output_writer_private = p;
/*
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index ad0fc6a04b..5074c8409f 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -488,6 +488,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
latestFlushPtr = GetStandbyFlushRecPtr(NULL);
if (remote_slot->confirmed_lsn > latestFlushPtr)
{
+ /*
+ * Can get here only if GUC 'standby_slot_names' on the primary server
+ * was not configured correctly.
+ */
ereport(AmLogicalSlotSyncWorkerProcess() ? LOG : ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("skipping slot synchronization as the received slot sync"
@@ -857,6 +861,13 @@ validate_remote_info(WalReceiverConn *wrconn)
remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull));
Assert(!isnull);
+ /*
+ * Slot sync is currently not supported on a cascading standby. This is
+ * because if we allow it, the primary server needs to wait for all the
+ * cascading standbys, otherwise, logical subscribers can still be ahead
+ * of one of the cascading standbys which we plan to promote. Thus, to
+ * avoid this additional complexity, we restrict it for the time being.
+ */
if (remote_in_recovery)
ereport(ERROR,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 2614f98ddd..f1d54a6981 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -46,13 +46,18 @@
#include "common/string.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/interrupt.h"
#include "replication/slotsync.h"
#include "replication/slot.h"
+#include "replication/walsender_private.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
+#include "utils/memutils.h"
+#include "utils/varlena.h"
/*
* Replication slot on-disk data structure.
@@ -115,10 +120,25 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
/* My backend's replication slot in the shared memory array */
ReplicationSlot *MyReplicationSlot = NULL;
-/* GUC variable */
+/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+/*
+ * This GUC lists streaming replication standby server slot names that
+ * logical WAL sender processes will wait for.
+ */
+char *standby_slot_names;
+
+/* This is parsed and cached list for raw standby_slot_names. */
+static List *standby_slot_names_list = NIL;
+
+/*
+ * Oldest LSN that has been confirmed to be flushed to the standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
+
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
@@ -2345,3 +2365,355 @@ GetSlotInvalidationCause(const char *conflict_reason)
Assert(found);
return result;
}
+
+/*
+ * A helper function to validate slots specified in GUC standby_slot_names.
+ */
+static bool
+validate_standby_slots(char **newval)
+{
+ char *rawname;
+ List *elemlist;
+ bool ok;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Verify syntax and parse string into a list of identifiers */
+ ok = SplitIdentifierString(rawname, ',', &elemlist);
+
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
+ else if (!ReplicationSlotCtl)
+ {
+ /*
+ * We cannot validate the replication slot if the replication slots'
+ * data has not been initialized. This is ok as we will anyway
+ * validate the specified slot when waiting for them to catch up. See
+ * StandbySlotsHaveCaughtup() for details.
+ */
+ }
+ else
+ {
+ /* Check that the specified slots exist and are logical slots */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, elemlist)
+ {
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ GUC_check_errdetail("replication slot \"%s\" does not exist",
+ name);
+ ok = false;
+ break;
+ }
+
+ if (!SlotIsPhysical(slot))
+ {
+ GUC_check_errdetail("\"%s\" is not a physical replication slot",
+ name);
+ ok = false;
+ break;
+ }
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+ }
+
+ pfree(rawname);
+ list_free(elemlist);
+ return ok;
+}
+
+/*
+ * GUC check_hook for standby_slot_names
+ */
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ if ((*newval)[0] == '\0')
+ return true;
+
+ /*
+ * "*" is not accepted as in that case the primary will not be able to
+ * know for which all standbys to wait for. It is inappropriate to block
+ * logical replication for physical slots that either lack associated
+ * standbys or have standbys associated that are not enabled for
+ * replication slot synchronization.
+ */
+ if (strcmp(*newval, "*") == 0)
+ {
+ GUC_check_errdetail("\"*\" is not accepted for standby_slot_names");
+ return false;
+ }
+
+ /* Now verify if the specified slots exist and have correct type */
+ if (!validate_standby_slots(newval))
+ return false;
+
+ *extra = guc_strdup(ERROR, *newval);
+
+ return true;
+}
+
+/*
+ * GUC assign_hook for standby_slot_names
+ */
+void
+assign_standby_slot_names(const char *newval, void *extra)
+{
+ List *standby_slots;
+ MemoryContext oldcxt;
+ char *standby_slot_names_cpy = extra;
+
+ /*
+ * The standby slots may have changed, so we must recompute the oldest
+ * LSN.
+ */
+ ss_oldest_flush_lsn = InvalidXLogRecPtr;
+
+ list_free(standby_slot_names_list);
+ standby_slot_names_list = NIL;
+
+ /* No value is specified for standby_slot_names. */
+ if (standby_slot_names_cpy == NULL)
+ return;
+
+ if (!SplitIdentifierString(standby_slot_names_cpy, ',', &standby_slots))
+ {
+ /* This should not happen if GUC checked check_standby_slot_names. */
+ elog(ERROR, "invalid list syntax");
+ }
+
+ /*
+ * Switch to the memory context under which GUC variables are allocated
+ * (GUCMemoryContext).
+ */
+ oldcxt = MemoryContextSwitchTo(GetMemoryChunkContext(standby_slot_names_cpy));
+ standby_slot_names_list = list_copy(standby_slots);
+ MemoryContextSwitchTo(oldcxt);
+}
+
+/*
+ * Return the standby_slot_names_list.
+ *
+ * Note that since we do not support syncing slots to cascading standbys, we
+ * return NIL if we are running in a standby to indicate that no standby slots
+ * need to be waited for.
+ */
+List *
+GetStandbySlotList(void)
+{
+ if (RecoveryInProgress())
+ return NIL;
+
+ return standby_slot_names_list;
+}
+
+/*
+ * Return true if the slots specified in standby_slot_names have caught up to
+ * the given WAL location, false otherwise.
+ *
+ * The elevel parameter specifies the error level used for logging messages
+ * related to slots that do not exist, are invalidated, or are inactive.
+ */
+bool
+StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel)
+{
+ int caught_up_slot_num = 0;
+ XLogRecPtr min_restart_lsn = InvalidXLogRecPtr;
+
+ /*
+ * Don't need to wait for the standbys to catch up if there is no value in
+ * standby_slot_names.
+ */
+ if (standby_slot_names_list == NIL)
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if we are on a standby
+ * server, since we do not support syncing slots to cascading standbys.
+ */
+ if (RecoveryInProgress())
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if they are already
+ * beyond the specified WAL location.
+ */
+ if (!XLogRecPtrIsInvalid(ss_oldest_flush_lsn) &&
+ ss_oldest_flush_lsn >= wait_for_lsn)
+ return true;
+
+ /*
+ * To prevent concurrent slot dropping and creation while filtering the
+ * slots, take the ReplicationSlotControlLock outside of the loop.
+ */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, standby_slot_names_list)
+ {
+ XLogRecPtr restart_lsn;
+ bool invalidated;
+ bool inactive;
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ /*
+ * If a slot name provided in standby_slot_names does not exist,
+ * report a message and exit the loop. A user can specify a slot
+ * name that does not exist just before the server startup. The
+ * GUC check_hook(validate_standby_slots) cannot validate such a
+ * slot during startup as the ReplicationSlotCtl shared memory is
+ * not initialized at that time. It is also possible for a user to
+ * drop the slot in standby_slot_names afterwards.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not exist",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider creating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (SlotIsLogical(slot))
+ {
+ /*
+ * If a logical slot name is provided in standby_slot_names,
+ * report a message and exit the loop. Similar to the non-existent
+ * case, a user can specify a logical slot name in
+ * standby_slot_names before the server startup, or drop an
+ * existing physical slot and recreate a logical slot with the
+ * same name.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot have logical replication slot \"%s\" in parameter %s",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting for correction on \"%s\".",
+ name),
+ errhint("Consider removing logical slot \"%s\" from parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ SpinLockAcquire(&slot->mutex);
+ restart_lsn = slot->data.restart_lsn;
+ invalidated = slot->data.invalidated != RS_INVAL_NONE;
+ inactive = slot->active_pid == 0;
+ SpinLockRelease(&slot->mutex);
+
+ if (invalidated)
+ {
+ /* Specified physical slot has been invalidated */
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("physical slot \"%s\" specified in parameter %s has been invalidated",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider dropping and recreating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (XLogRecPtrIsInvalid(restart_lsn) || restart_lsn < wait_for_lsn)
+ {
+ /* Log a message if no active_pid for this physical slot */
+ if (inactive)
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not have active_pid",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider starting standby associated with \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+
+ /* Continue if the current slot hasn't caught up. */
+ break;
+ }
+
+ Assert(restart_lsn >= wait_for_lsn);
+
+ if (XLogRecPtrIsInvalid(min_restart_lsn) ||
+ min_restart_lsn > restart_lsn)
+ min_restart_lsn = restart_lsn;
+
+ caught_up_slot_num++;
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /*
+ * Return false if not all the standbys have caught up to the specified
+ * WAL location.
+ */
+ if (caught_up_slot_num != list_length(standby_slot_names_list))
+ return false;
+
+ /* The ss_oldest_flush_lsn must not retreat. */
+ Assert(XLogRecPtrIsInvalid(ss_oldest_flush_lsn) ||
+ min_restart_lsn >= ss_oldest_flush_lsn);
+
+ ss_oldest_flush_lsn = min_restart_lsn;
+
+ return true;
+}
+
+/*
+ * Wait for physical standbys to confirm receiving the given lsn.
+ *
+ * Used by logical decoding SQL functions. It waits for physical standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+void
+WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
+{
+ /*
+ * Don't need to wait for the standby to catch up if the current acquired
+ * slot is not a logical failover slot, or there is no value in
+ * standby_slot_names.
+ */
+ if (!MyReplicationSlot->data.failover || standby_slot_names_list == NIL)
+ return;
+
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+
+ for (;;)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /* Exit if done waiting for every slot. */
+ if (StandbySlotsHaveCaughtup(wait_for_lsn, WARNING))
+ break;
+
+ /*
+ * Wait for the slots in the standby_slot_names to catch up, but use a
+ * timeout (1s) so we can also check if the standby_slot_names has
+ * been changed.
+ */
+ ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000,
+ WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION);
+ }
+
+ ConditionVariableCancelSleep();
+}
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 768a304723..ad79e1fccd 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -464,6 +464,12 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* crash, but this makes the data consistent after a clean shutdown.
*/
ReplicationSlotMarkDirty();
+
+ /*
+ * Wake up logical walsenders holding logical failover slots after
+ * updating the restart_lsn of the physical slot.
+ */
+ PhysicalWakeupLogicalWalSnd();
}
return retlsn;
@@ -504,6 +510,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto)
.segment_close = wal_segment_close),
NULL, NULL, NULL);
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to moveto lsn.
+ */
+ WaitForStandbyConfirmation(moveto);
+
/*
* Start reading at the slot's restart_lsn, which we know to point to
* a valid record.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 0f1047179c..e3c5cb4a90 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1725,26 +1725,113 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
ProcessPendingWrites();
}
+/*
+ * Wake up the logical walsender processes with logical failover slots if the
+ * currently acquired physical slot is specified in standby_slot_names GUC.
+ */
+void
+PhysicalWakeupLogicalWalSnd(void)
+{
+ List *standby_slots;
+
+ Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot));
+
+ standby_slots = GetStandbySlotList();
+
+ foreach_ptr(char, name, standby_slots)
+ {
+ if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0)
+ {
+ ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv);
+ return;
+ }
+ }
+}
+
+/*
+ * Returns true if not all standbys have caught up to the flushed position
+ * (flushed_lsn) when the current acquired slot is a logical failover
+ * slot and we are streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForStandbys(XLogRecPtr flushed_lsn, uint32 *wait_event)
+{
+ int elevel = got_STOPPING ? ERROR : WARNING;
+ bool failover_slot;
+
+ failover_slot = (replication_active && MyReplicationSlot->data.failover);
+
+ /*
+ * Note that after receiving the shutdown signal, an ERROR is reported if
+ * any slots are dropped, invalidated, or inactive. This measure is taken
+ * to prevent the walsender from waiting indefinitely.
+ */
+ if (failover_slot && !StandbySlotsHaveCaughtup(flushed_lsn, elevel))
+ {
+ *wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
+ return true;
+ }
+
+ *wait_event = 0;
+ return false;
+}
+
+/*
+ * Returns true if we need to wait for WALs to be flushed to disk, or if not
+ * all standbys have caught up to the flushed position (flushed_lsn) when the
+ * current acquired slot is a logical failover slot and we are
+ * streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
+{
+ /* Check if we need to wait for WALs to be flushed to disk */
+ if (target_lsn > flushed_lsn)
+ {
+ *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
+ return true;
+ }
+
+ /* Check if the standby slots have caught up to the flushed position */
+ return NeedToWaitForStandbys(flushed_lsn, wait_event);
+}
+
/*
* Wait till WAL < loc is flushed to disk so it can be safely sent to client.
*
- * Returns end LSN of flushed WAL. Normally this will be >= loc, but
- * if we detect a shutdown request (either from postmaster or client)
- * we will return early, so caller must always check.
+ * If the walsender holds a logical slot that has enabled failover, we also
+ * wait for all the specified streaming replication standby servers to confirm
+ * receipt of WAL up to RecentFlushPtr. It is beneficial to wait here for the
+ * confirmation up to RecentFlushPtr rather than waiting before transmitting
+ * each change to logical subscribers, which is already covered by
+ * RecentFlushPtr.
+ *
+ * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we
+ * detect a shutdown request (either from postmaster or client) we will return
+ * early, so caller must always check.
*/
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
int wakeEvents;
+ uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
- * have enough WAL available. This is particularly interesting if we're
- * far behind.
+ * have enough WAL available and all the standby servers have confirmed
+ * receipt of WAL up to RecentFlushPtr. This is particularly interesting
+ * if we're far behind.
*/
- if (RecentFlushPtr != InvalidXLogRecPtr &&
- loc <= RecentFlushPtr)
+ if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
return RecentFlushPtr;
/* Get a more recent flush pointer. */
@@ -1753,8 +1840,14 @@ WalSndWaitForWal(XLogRecPtr loc)
else
RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * Within the loop, we wait for the necessary WALs to be flushed to disk
+ * first, followed by waiting for standbys to catch up if there are enough
+ * WALs (see NeedToWaitForWal()) or upon receiving the shutdown signal.
+ */
for (;;)
{
+ bool wait_for_standby_at_stop = false;
long sleeptime;
/* Clear any already-pending wakeups */
@@ -1781,21 +1874,35 @@ WalSndWaitForWal(XLogRecPtr loc)
if (got_STOPPING)
XLogBackgroundFlush();
- /* Update our idea of the currently flushed position. */
- if (!RecoveryInProgress())
- RecentFlushPtr = GetFlushRecPtr(NULL);
- else
- RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * To avoid the scenario where standbys need to catch up to a newer WAL
+ * location in each iteration, we update our idea of the currently
+ * flushed position only if we are not waiting for standbys to catch
+ * up.
+ */
+ if (wait_event != WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ {
+ if (!RecoveryInProgress())
+ RecentFlushPtr = GetFlushRecPtr(NULL);
+ else
+ RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ }
/*
- * If postmaster asked us to stop, don't wait anymore.
+ * If postmaster asked us to stop and the standby slots have caught up
+ * to the flushed position, don't wait anymore.
*
* It's important to do this check after the recomputation of
* RecentFlushPtr, so we can send all remaining data before shutting
* down.
*/
if (got_STOPPING)
- break;
+ {
+ if (NeedToWaitForStandbys(RecentFlushPtr, &wait_event))
+ wait_for_standby_at_stop = true;
+ else
+ break;
+ }
/*
* We only send regular messages to the client for full decoded
@@ -1810,11 +1917,18 @@ WalSndWaitForWal(XLogRecPtr loc)
!waiting_for_ping_response)
WalSndKeepalive(false, InvalidXLogRecPtr);
- /* check whether we're done */
- if (loc <= RecentFlushPtr)
+ /*
+ * Exit the loop if already caught up and doesn't need to wait for
+ * standby slots.
+ */
+ if (!wait_for_standby_at_stop &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
break;
- /* Waiting for new WAL. Since we need to wait, we're now caught up. */
+ /*
+ * Waiting for new WAL or waiting for standbys to catch up. Since we
+ * need to wait, we're now caught up.
+ */
WalSndCaughtUp = true;
/*
@@ -1852,7 +1966,9 @@ WalSndWaitForWal(XLogRecPtr loc)
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL);
+ Assert(wait_event != 0);
+
+ WalSndWait(wakeEvents, sleeptime, wait_event);
}
/* reactivate latch so WalSndLoop knows to continue */
@@ -2262,6 +2378,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn)
{
ReplicationSlotMarkDirty();
ReplicationSlotsComputeRequiredLSN();
+ PhysicalWakeupLogicalWalSnd();
}
/*
@@ -3535,6 +3652,7 @@ WalSndShmemInit(void)
ConditionVariableInit(&WalSndCtl->wal_flush_cv);
ConditionVariableInit(&WalSndCtl->wal_replay_cv);
+ ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv);
}
}
@@ -3604,8 +3722,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event)
*
* And, we use separate shared memory CVs for physical and logical
* walsenders for selective wake ups, see WalSndWakeup() for more details.
+ *
+ * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV
+ * until awakened by physical walsenders after the walreceiver confirms
+ * the receipt of the LSN.
*/
- if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
+ if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+ else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv);
else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index ec2f31f82a..c08e00d1d6 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP
LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server."
LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server."
SSL_OPEN_SERVER "Waiting for SSL while attempting connection."
+WAIT_FOR_STANDBY_CONFIRMATION "Waiting for WAL to be received and flushed by the physical standby."
WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process."
WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 45013582a7..d77214795d 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4670,6 +4670,20 @@ struct config_string ConfigureNamesString[] =
check_debug_io_direct, assign_debug_io_direct, NULL
},
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("Lists streaming replication standby server slot "
+ "names that logical WAL sender processes will wait for."),
+ gettext_noop("Logical WAL sender processes will send decoded "
+ "changes to plugins only after the specified "
+ "replication slots confirm receiving WAL."),
+ GUC_LIST_INPUT
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, assign_standby_slot_names, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index edcc0282b2..2244ee52f7 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -343,6 +343,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsender processes will wait for
# - Standby Servers -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index acbf567150..61a6b97952 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -226,6 +226,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
+extern PGDLLIMPORT char *standby_slot_names;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -274,4 +275,8 @@ extern void CheckSlotPermissions(void);
extern ReplicationSlotInvalidationCause
GetSlotInvalidationCause(const char *conflict_reason);
+extern List *GetStandbySlotList(void);
+extern bool StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel);
+extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn);
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index 0c3996e926..f2d8297f01 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -39,6 +39,7 @@ extern void InitWalSender(void);
extern bool exec_replication_command(const char *cmd_string);
extern void WalSndErrorCleanup(void);
extern void WalSndResourceCleanup(bool isCommit);
+extern void PhysicalWakeupLogicalWalSnd(void);
extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
extern void WalSndSignals(void);
extern Size WalSndShmemSize(void);
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 3113e9ea47..109924ffcd 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -113,6 +113,13 @@ typedef struct
ConditionVariable wal_flush_cv;
ConditionVariable wal_replay_cv;
+ /*
+ * Used by physical walsenders holding slots specified in
+ * standby_slot_names to wake up logical walsenders holding logical
+ * failover slots when a walreceiver confirms the receipt of LSN.
+ */
+ ConditionVariable wal_confirm_rcv_cv;
+
WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER];
} WalSndCtlData;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index c8a7aa9a11..d64dc5fcdb 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra,
extern void assign_wal_consistency_checking(const char *newval, void *extra);
extern bool check_wal_segment_size(int *newval, void **extra, GucSource source);
extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
+extern bool check_standby_slot_names(char **newval, void **extra,
+ GucSource source);
+extern void assign_standby_slot_names(const char *newval, void *extra);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl
index 5c7b4ca5e3..b95d95c06f 100644
--- a/src/test/recovery/t/006_logical_decoding.pl
+++ b/src/test/recovery/t/006_logical_decoding.pl
@@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'},
undef, 'logical slot was actually dropped with DB');
# Test logical slot advancing and its durability.
+# Passing failover=true (last arg) should not have any impact on advancing.
my $logical_slot = 'logical_slot';
$node_primary->safe_psql('postgres',
- "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);"
+ "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);"
);
$node_primary->psql(
'postgres', "
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 021c58f621..99025b352a 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -504,18 +504,258 @@ ok( $standby1->poll_query_until(
"SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot' AND synced AND NOT temporary;"),
'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby');
+##################################################
+# Test that logical failover replication slots wait for the specified
+# physical replication slots to receive the changes first. It uses the
+# following set up:
+#
+# (physical standbys)
+# | ----> standby1 (primary_slot_name = sb1_slot)
+# | ----> standby2 (primary_slot_name = sb2_slot)
+# primary ----- |
+# (logical replication)
+# | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
+# | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
+#
+# standby_slot_names = 'sb1_slot'
+#
+# The setup is configured in such a way that the logical slot of subscriber1 is
+# enabled for failover, and thus the subscriber1 will wait for the physical
+# slot of standby1(sb1_slot) to catch up before receiving the decoded changes.
+##################################################
+
+$backup_name = 'backup3';
+
+$primary->psql('postgres',
+ q{SELECT pg_create_physical_replication_slot('sb2_slot');});
+
+$primary->backup($backup_name);
+
+# Create another standby
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup(
+ $primary, $backup_name,
+ has_streaming => 1,
+ has_restoring => 1);
+$standby2->append_conf(
+ 'postgresql.conf', qq(
+primary_slot_name = 'sb2_slot'
+));
+$standby2->start;
+$primary->wait_for_replay_catchup($standby2);
+
+# Configure primary to disallow any logical slots that have enabled failover
+# from getting ahead of the specified physical replication slot (sb1_slot).
+$primary->append_conf(
+ 'postgresql.conf', qq(
+standby_slot_names = 'sb1_slot'
+));
+$primary->reload;
+
+# Create another subscriber node without enabling failover, wait for sync to
+# complete
+my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$subscriber2->init;
+$subscriber2->start;
+$subscriber2->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE tab_int (a int PRIMARY KEY);
+ CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot);
+]);
+
+$subscriber2->wait_for_subscription_sync;
+
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+my $offset = -s $primary->logfile;
+
+# Stop the standby associated with the specified physical replication slot
+# (sb1_slot) so that the logical replication slot (lsub1_slot) won't receive
+# changes until the standby comes up.
+$standby1->stop;
+
+# Create some data on the primary
+my $primary_row_count = 20;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(11, $primary_row_count);");
+
+# Wait until the standby2 that's still running gets the data from the primary
+$primary->wait_for_replay_catchup($standby2);
+$result = $standby2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby2 gets data from primary");
+
+# Wait for regress_mysub2 to get the data from the primary. This subscription
+# was not enabled for failover so it gets the data without waiting for any
+# standbys.
+$primary->wait_for_catchup('regress_mysub2');
+$result = $subscriber2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "subscriber2 gets data from primary");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# The regress_mysub1 was enabled for failover so it doesn't get the data from
+# primary and keeps waiting for the standby specified in standby_slot_names
+# (sb1_slot aka standby1).
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) <> $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data from primary until standby1 acknowledges changes"
+);
+
+# Start the standby specified in standby_slot_names (sb1_slot aka standby1) and
+# wait for it to catch up with the primary.
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+$result = $standby1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby1 gets data from primary");
+
+# Now that the standby specified in standby_slot_names is up and running, the
+# primary can send the decoded changes to the subscription enabled for failover
+# (i.e. regress_mysub1). While the standby was down, regress_mysub1 didn't
+# receive any data from the primary. i.e. the primary didn't allow it to go
+# ahead of standby.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 acknowledges changes");
+
+##################################################
+# Verify that when using pg_logical_slot_get_changes to consume changes from a
+# logical slot with failover enabled, it will also wait for the slots specified
+# in standby_slot_names to catch up.
+##################################################
+
+# Stop the standby associated with the specified physical replication slot so
+# that the logical replication slot won't receive changes until the standby
+# slot's restart_lsn is advanced or the slot is removed from the
+# standby_slot_names list.
+$primary->safe_psql('postgres', "TRUNCATE tab_int;");
+$primary->wait_for_catchup('regress_mysub1');
+$standby1->stop;
+
+# Disable the regress_mysub1 to prevent the logical walsender from generating
+# more warnings.
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 DISABLE");
+
+# Wait for the replication slot to become inactive on the publisher
+$primary->poll_query_until(
+ 'postgres',
+ "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'",
+ 1);
+
+# Create a logical 'test_decoding' replication slot with failover enabled
+$primary->safe_psql('postgres',
+ "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);"
+);
+
+my $back_q = $primary->background_psql(
+ 'postgres',
+ on_error_stop => 0,
+ timeout => $PostgreSQL::Test::Utils::timeout_default);
+
+# pg_logical_slot_get_changes will be blocked until the standby catches up,
+# hence it needs to be executed in a background session.
+$offset = -s $primary->logfile;
+$back_q->query_until(
+ qr/logical_slot_get_changes/, q(
+ \echo logical_slot_get_changes
+ SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);
+));
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the function
+# pg_logical_slot_get_changes should now return, and the session can be
+# stopped.
+$back_q->quit;
+
+$primary->safe_psql('postgres',
+ "SELECT pg_drop_replication_slot('test_slot');"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
+# Enable the regress_mysub1 for further tests
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+##################################################
+# Test that logical replication will wait for the user-created inactive
+# physical slot to catch up until we remove the slot from standby_slot_names.
+##################################################
+
+$offset = -s $primary->logfile;
+
+# Create some data on the primary
+$primary_row_count = 10;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# The regress_mysub1 doesn't get the data from primary because the specified
+# standby slot (sb1_slot) in standby_slot_names is inactive.
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data as the sb1_slot doesn't catch up");
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the primary server should now
+# send the decoded changes to the subscription.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
##################################################
# Promote the standby1 to primary. Confirm that:
# a) the slot 'lsub1_slot' is retained on the new primary
# b) logical replication for regress_mysub1 is resumed successfully after failover
##################################################
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+
$standby1->promote;
# Update subscription with the new primary's connection info
my $standby1_conninfo = $standby1->connstr . ' dbname=postgres';
$subscriber1->safe_psql('postgres',
- "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo';
- ALTER SUBSCRIPTION regress_mysub1 ENABLE; ");
+ "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo';");
# Confirm the synced slot 'lsub1_slot' is retained on the new primary
is($standby1->safe_psql('postgres',
--
2.30.0.windows.2
[application/octet-stream] v106-0002-Document-the-steps-to-check-if-the-standby-is-r.patch (7.0K, ../../OS0PR01MB571613E2CA15D369D943EF4394222@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v106-0002-Document-the-steps-to-check-if-the-standby-is-r.patch)
download | inline diff:
From 6fff865175135efa3c0ea0432715b603ab245cf5 Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Fri, 19 Jan 2024 11:04:16 +0530
Subject: [PATCH v106 2/2] Document the steps to check if the standby is ready
for failover
---
doc/src/sgml/high-availability.sgml | 9 ++
doc/src/sgml/logical-replication.sgml | 136 ++++++++++++++++++++++++++
2 files changed, 145 insertions(+)
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 236c0af65f..36215aa68c 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -1487,6 +1487,15 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)'
Written administration procedures are advised.
</para>
+ <para>
+ If you have opted for synchronization of logical slots (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>),
+ then before switching to the standby server, it is recommended to check
+ if the logical slots synchronized on the standby server are ready
+ for failover. This can be done by following the steps described in
+ <xref linkend="logical-replication-failover"/>.
+ </para>
+
<para>
To trigger failover of a log-shipping standby server, run
<command>pg_ctl promote</command> or call <function>pg_promote()</function>.
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ec2130669e..be59d306a1 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -687,6 +687,142 @@ ALTER SUBSCRIPTION
</sect1>
+ <sect1 id="logical-replication-failover">
+ <title>Logical Replication Failover</title>
+
+ <para>
+ When the publisher server is the primary server of a streaming replication,
+ the logical slots on that primary server can be synchronized to the standby
+ server by specifying <literal>failover = true</literal> when creating
+ subscriptions for those publications. Enabling failover ensures a seamless
+ transition of those subscriptions after the standby is promoted. They can
+ continue subscribing to publications now on the new primary server without
+ any data loss.
+ </para>
+
+ <para>
+ Because the slot synchronization logic copies asynchronously, it is
+ necessary to confirm that replication slots have been synced to the standby
+ server before the failover happens. Furthermore, to ensure a successful
+ failover, the standby server must not be lagging behind the subscriber. It
+ is highly recommended to use
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ to prevent the subscriber from consuming changes faster than the hot standby.
+ To confirm that the standby server is indeed ready for failover, follow
+ these 2 steps:
+ </para>
+
+ <procedure>
+ <step performance="required">
+ <para>
+ Confirm that all the necessary logical replication slots have been synced to
+ the standby server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node, use the following SQL to identify
+ which slots should be synced to the standby that we plan to promote.
+<programlisting>
+test_sub=# SELECT
+ array_agg(slotname) AS slots
+ FROM
+ ((
+ SELECT r.srsubid AS subid, CONCAT('pg_', srsubid, '_sync_', srrelid, '_', ctl.system_identifier) AS slotname
+ FROM pg_control_system() ctl, pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate = 'f' AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT s.oid AS subid, s.subslotname as slotname
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ slots
+-------
+ {sub1,sub2,sub3}
+(1 row)
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, check that the logical replication slots identified above exist on
+ the standby server and are ready for failover.
+<programlisting>
+test_standby=# SELECT slot_name, (synced AND NOT temporary AND conflict_reason IS NULL) AS failover_ready
+ FROM pg_replication_slots
+ WHERE slot_name IN ('sub1','sub2','sub3');
+ slot_name | failover_ready
+-------------+----------------
+ sub1 | t
+ sub2 | t
+ sub3 | t
+(3 rows)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+
+ <step performance="required">
+ <para>
+ Confirm that the standby server is not lagging behind the subscribers.
+ This step can be skipped if
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ has been correctly configured. If standby_slot_names is not configured
+ correctly, it is highly recommended to run this step after the primary
+ server is down, otherwise the results of the query may vary at different
+ points of time due to the ongoing replication on the logical subscribers
+ from the primary server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node check the last replayed WAL.
+ This step needs to be run on the database(s) that includes the failover
+ enabled subscription(s), to find the last replayed WAL on each database.
+<programlisting>
+test_sub=# SELECT
+ MAX(remote_lsn) AS remote_lsn_on_subscriber
+ FROM
+ ((
+ SELECT (CASE WHEN r.srsubstate = 'f' THEN pg_replication_origin_progress(CONCAT('pg_', r.srsubid, '_', r.srrelid), false)
+ WHEN r.srsubstate IN ('s', 'r') THEN r.srsublsn END) AS remote_lsn
+ FROM pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate IN ('f', 's', 'r') AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT pg_replication_origin_progress(CONCAT('pg_', s.oid), false) AS remote_lsn
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ remote_lsn_on_subscriber
+--------------------------
+ 0/3000388
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, on the standby server check that the last-received WAL location
+ is ahead of the replayed WAL location(s) on the subscriber identified
+ above. If the above SQL result was NULL, it means the subscriber has not
+ yet replayed any WAL, so the standby server must be ahead of the
+ subscriber, and this step can be skipped.
+<programlisting>
+test_standby=# SELECT pg_last_wal_receive_lsn() >= '0/3000388'::pg_lsn AS failover_ready;
+ failover_ready
+----------------
+ t
+(1 row)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+ </procedure>
+
+ <para>
+ If the result (<literal>failover_ready</literal>) of both above steps is
+ true, existing subscriptions will be able to continue without data loss.
+ </para>
+
+ </sect1>
+
<sect1 id="logical-replication-row-filter">
<title>Row Filters</title>
--
2.30.0.windows.2
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-06 01:30 Masahiko Sawada <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Masahiko Sawada @ 2024-03-06 01:30 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Fri, Mar 1, 2024 at 4:21 PM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> On Friday, March 1, 2024 2:11 PM Masahiko Sawada <[email protected]> wrote:
> >
> >
> > ---
> > +void
> > +assign_standby_slot_names(const char *newval, void *extra) {
> > + List *standby_slots;
> > + MemoryContext oldcxt;
> > + char *standby_slot_names_cpy = extra;
> > +
> >
> > Given that the newval and extra have the same data (standby_slot_names
> > value), why do we not use newval instead? I think that if we use
> > newval, we don't need to guc_strdup() in check_standby_slot_names(),
> > we might need to do list_copy_deep() instead, though. It's not clear
> > to me as there is no comment.
>
> I think SplitIdentifierString will modify the passed in string, so we'd better
> not pass the newval to it, otherwise the stored guc string(standby_slot_names)
> will be changed. I can see we are doing similar thing in other GUC check/assign
> function as well. (check_wal_consistency_checking/
> assign_wal_consistency_checking, check_createrole_self_grant/
> assign_createrole_self_grant ...).
Why does it have to be a List in the first place? In earlier version
patches, we used to copy the list and delete the element until it
became empty, while waiting for physical wal senders. But we now just
refer to each slot name in the list. The current code assumes that
stnadby_slot_names_cpy is allocated in GUCMemoryContext but once it
changes, it will silently get broken. I think we can check and assign
standby_slot_names in a similar way to check/assign_temp_tablespaces
and check/assign_synchronous_standby_names.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-06 03:03 Zhijie Hou (Fujitsu) <[email protected]>
parent: Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-06 03:03 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Wednesday, March 6, 2024 9:30 AM Masahiko Sawada <[email protected]> wrote:
Hi,
> On Fri, Mar 1, 2024 at 4:21 PM Zhijie Hou (Fujitsu) <[email protected]>
> wrote:
> >
> > On Friday, March 1, 2024 2:11 PM Masahiko Sawada
> <[email protected]> wrote:
> > >
> > >
> > > ---
> > > +void
> > > +assign_standby_slot_names(const char *newval, void *extra) {
> > > + List *standby_slots;
> > > + MemoryContext oldcxt;
> > > + char *standby_slot_names_cpy = extra;
> > > +
> > >
> > > Given that the newval and extra have the same data
> > > (standby_slot_names value), why do we not use newval instead? I
> > > think that if we use newval, we don't need to guc_strdup() in
> > > check_standby_slot_names(), we might need to do list_copy_deep()
> > > instead, though. It's not clear to me as there is no comment.
> >
> > I think SplitIdentifierString will modify the passed in string, so
> > we'd better not pass the newval to it, otherwise the stored guc
> > string(standby_slot_names) will be changed. I can see we are doing
> > similar thing in other GUC check/assign function as well.
> > (check_wal_consistency_checking/ assign_wal_consistency_checking,
> > check_createrole_self_grant/ assign_createrole_self_grant ...).
>
> Why does it have to be a List in the first place?
I thought the List type is convenient to use here, as we have existing list
build function(SplitIdentifierString), and have convenient list macro to loop
the list(foreach_ptr) which can save some codes.
> In earlier version patches, we
> used to copy the list and delete the element until it became empty, while
> waiting for physical wal senders. But we now just refer to each slot name in the
> list. The current code assumes that stnadby_slot_names_cpy is allocated in
> GUCMemoryContext but once it changes, it will silently get broken. I think we
> can check and assign standby_slot_names in a similar way to
> check/assign_temp_tablespaces and
> check/assign_synchronous_standby_names.
Yes, we could do follow it by allocating an array and copy each slot name into
it, but it also requires some codes to build and scan the array. So, is it possible
to expose the GucMemorycontext or have an API like guc_copy_list instead ?
If we don't want to touch the guc api, I am ok with using an array as well.
Best Regards,
Hou zj
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-06 06:37 Masahiko Sawada <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Masahiko Sawada @ 2024-03-06 06:37 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; Amit Kapila <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Fri, Mar 1, 2024 at 3:22 PM Peter Smith <[email protected]> wrote:
>
> On Fri, Mar 1, 2024 at 5:11 PM Masahiko Sawada <[email protected]> wrote:
> >
> ...
> > + /*
> > + * "*" is not accepted as in that case primary will not be able to know
> > + * for which all standbys to wait for. Even if we have physical slots
> > + * info, there is no way to confirm whether there is any standby
> > + * configured for the known physical slots.
> > + */
> > + if (strcmp(*newval, "*") == 0)
> > + {
> > + GUC_check_errdetail("\"*\" is not accepted for
> > standby_slot_names");
> > + return false;
> > + }
> >
> > Why only '*' is checked aside from validate_standby_slots()? I think
> > that the doc doesn't mention anything about '*' and '*' cannot be used
> > as a replication slot name. So even if we don't have this check, it
> > might be no problem.
> >
>
> Hi, a while ago I asked this same question. See [1 #28] for the response..
Thanks. Quoting the response from the email:
SplitIdentifierString() does not give error for '*' and '*' can be considered
as valid value which if accepted can mislead user that all the standbys's slots
are now considered, which is not the case here. So we want to explicitly call
out this case i.e. '*' is not accepted as valid value for standby_slot_names.
IIUC we're concerned with a case like where the user confused
standby_slot_names values with synchronous_standby_names values. Which
means we would need to keep thath check consistent with available
values of synchronous_standby_names. For example, if we support a
regexp for synchronous_standby_names, we will have to update the check
so we disallow other special characters. Also, if we add a new
replication-related parameter that accepts other special characters as
the value in the future, will we want to raise an error also for such
values in check_standby_slot_names()?
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-06 08:53 Amit Kapila <[email protected]>
parent: Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Amit Kapila @ 2024-03-06 08:53 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Peter Smith <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Wed, Mar 6, 2024 at 12:07 PM Masahiko Sawada <[email protected]> wrote:
>
> On Fri, Mar 1, 2024 at 3:22 PM Peter Smith <[email protected]> wrote:
> >
> > On Fri, Mar 1, 2024 at 5:11 PM Masahiko Sawada <[email protected]> wrote:
> > >
> > ...
> > > + /*
> > > + * "*" is not accepted as in that case primary will not be able to know
> > > + * for which all standbys to wait for. Even if we have physical slots
> > > + * info, there is no way to confirm whether there is any standby
> > > + * configured for the known physical slots.
> > > + */
> > > + if (strcmp(*newval, "*") == 0)
> > > + {
> > > + GUC_check_errdetail("\"*\" is not accepted for
> > > standby_slot_names");
> > > + return false;
> > > + }
> > >
> > > Why only '*' is checked aside from validate_standby_slots()? I think
> > > that the doc doesn't mention anything about '*' and '*' cannot be used
> > > as a replication slot name. So even if we don't have this check, it
> > > might be no problem.
> > >
> >
> > Hi, a while ago I asked this same question. See [1 #28] for the response..
>
> Thanks. Quoting the response from the email:
>
> SplitIdentifierString() does not give error for '*' and '*' can be considered
> as valid value which if accepted can mislead user that all the standbys's slots
> are now considered, which is not the case here. So we want to explicitly call
> out this case i.e. '*' is not accepted as valid value for standby_slot_names.
>
> IIUC we're concerned with a case like where the user confused
> standby_slot_names values with synchronous_standby_names values. Which
> means we would need to keep thath check consistent with available
> values of synchronous_standby_names.
>
Both have different formats to specify. For example, for
synchronous_standby_names we have the following kind of syntax to
specify:
[FIRST] num_sync ( standby_name [, ...] )
ANY num_sync ( standby_name [, ...] )
standby_name [, ...]
I don't think we can have a common check for both of them as the
specifications are different. In fact, I don't think we need a special
check for '*'. The user will anyway get a WARNING at a later point
that the replication slot with that name doesn't exist.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-06 13:13 Zhijie Hou (Fujitsu) <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-06 13:13 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Wednesday, March 6, 2024 11:04 AM Zhijie Hou (Fujitsu) <[email protected]> wrote:
>
> On Wednesday, March 6, 2024 9:30 AM Masahiko Sawada
> <[email protected]> wrote:
>
> Hi,
>
> > On Fri, Mar 1, 2024 at 4:21 PM Zhijie Hou (Fujitsu)
> > <[email protected]>
> > wrote:
> > >
> > > On Friday, March 1, 2024 2:11 PM Masahiko Sawada
> > <[email protected]> wrote:
> > > >
> > > >
> > > > ---
> > > > +void
> > > > +assign_standby_slot_names(const char *newval, void *extra) {
> > > > + List *standby_slots;
> > > > + MemoryContext oldcxt;
> > > > + char *standby_slot_names_cpy = extra;
> > > > +
> > > >
> > > > Given that the newval and extra have the same data
> > > > (standby_slot_names value), why do we not use newval instead? I
> > > > think that if we use newval, we don't need to guc_strdup() in
> > > > check_standby_slot_names(), we might need to do list_copy_deep()
> > > > instead, though. It's not clear to me as there is no comment.
> > >
> > > I think SplitIdentifierString will modify the passed in string, so
> > > we'd better not pass the newval to it, otherwise the stored guc
> > > string(standby_slot_names) will be changed. I can see we are doing
> > > similar thing in other GUC check/assign function as well.
> > > (check_wal_consistency_checking/ assign_wal_consistency_checking,
> > > check_createrole_self_grant/ assign_createrole_self_grant ...).
> >
> > Why does it have to be a List in the first place?
>
> I thought the List type is convenient to use here, as we have existing list build
> function(SplitIdentifierString), and have convenient list macro to loop the
> list(foreach_ptr) which can save some codes.
>
> > In earlier version patches, we
> > used to copy the list and delete the element until it became empty,
> > while waiting for physical wal senders. But we now just refer to each
> > slot name in the list. The current code assumes that
> > stnadby_slot_names_cpy is allocated in GUCMemoryContext but once it
> > changes, it will silently get broken. I think we can check and assign
> > standby_slot_names in a similar way to check/assign_temp_tablespaces
> > and check/assign_synchronous_standby_names.
>
> Yes, we could do follow it by allocating an array and copy each slot name into it,
> but it also requires some codes to build and scan the array. So, is it possible to
> expose the GucMemorycontext or have an API like guc_copy_list instead ?
> If we don't want to touch the guc api, I am ok with using an array as well.
I rethink about this and realize that it's not good to do the memory allocation
in assign hook function. As the "src/backend/utils/misc/README" said, we'd better
do that in check hook function and pass it via extra to assign hook function. And thus
array is a good choice in this case rather than a List which cannot be passed to *extra.
Here is the V107 patch set which parse and cache the standby slot names in an
array instead of a List.
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v107-0002-Document-the-steps-to-check-if-the-standby-is-r.patch (7.0K, ../../OS0PR01MB57161AEDBD6D3538905E069994212@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v107-0002-Document-the-steps-to-check-if-the-standby-is-r.patch)
download | inline diff:
From 94ed723efb005142940925af70fa577c3877c732 Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Fri, 19 Jan 2024 11:04:16 +0530
Subject: [PATCH v107 2/2] Document the steps to check if the standby is ready
for failover
---
doc/src/sgml/high-availability.sgml | 9 ++
doc/src/sgml/logical-replication.sgml | 136 ++++++++++++++++++++++++++
2 files changed, 145 insertions(+)
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 236c0af65f..36215aa68c 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -1487,6 +1487,15 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)'
Written administration procedures are advised.
</para>
+ <para>
+ If you have opted for synchronization of logical slots (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>),
+ then before switching to the standby server, it is recommended to check
+ if the logical slots synchronized on the standby server are ready
+ for failover. This can be done by following the steps described in
+ <xref linkend="logical-replication-failover"/>.
+ </para>
+
<para>
To trigger failover of a log-shipping standby server, run
<command>pg_ctl promote</command> or call <function>pg_promote()</function>.
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ec2130669e..be59d306a1 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -687,6 +687,142 @@ ALTER SUBSCRIPTION
</sect1>
+ <sect1 id="logical-replication-failover">
+ <title>Logical Replication Failover</title>
+
+ <para>
+ When the publisher server is the primary server of a streaming replication,
+ the logical slots on that primary server can be synchronized to the standby
+ server by specifying <literal>failover = true</literal> when creating
+ subscriptions for those publications. Enabling failover ensures a seamless
+ transition of those subscriptions after the standby is promoted. They can
+ continue subscribing to publications now on the new primary server without
+ any data loss.
+ </para>
+
+ <para>
+ Because the slot synchronization logic copies asynchronously, it is
+ necessary to confirm that replication slots have been synced to the standby
+ server before the failover happens. Furthermore, to ensure a successful
+ failover, the standby server must not be lagging behind the subscriber. It
+ is highly recommended to use
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ to prevent the subscriber from consuming changes faster than the hot standby.
+ To confirm that the standby server is indeed ready for failover, follow
+ these 2 steps:
+ </para>
+
+ <procedure>
+ <step performance="required">
+ <para>
+ Confirm that all the necessary logical replication slots have been synced to
+ the standby server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node, use the following SQL to identify
+ which slots should be synced to the standby that we plan to promote.
+<programlisting>
+test_sub=# SELECT
+ array_agg(slotname) AS slots
+ FROM
+ ((
+ SELECT r.srsubid AS subid, CONCAT('pg_', srsubid, '_sync_', srrelid, '_', ctl.system_identifier) AS slotname
+ FROM pg_control_system() ctl, pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate = 'f' AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT s.oid AS subid, s.subslotname as slotname
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ slots
+-------
+ {sub1,sub2,sub3}
+(1 row)
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, check that the logical replication slots identified above exist on
+ the standby server and are ready for failover.
+<programlisting>
+test_standby=# SELECT slot_name, (synced AND NOT temporary AND conflict_reason IS NULL) AS failover_ready
+ FROM pg_replication_slots
+ WHERE slot_name IN ('sub1','sub2','sub3');
+ slot_name | failover_ready
+-------------+----------------
+ sub1 | t
+ sub2 | t
+ sub3 | t
+(3 rows)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+
+ <step performance="required">
+ <para>
+ Confirm that the standby server is not lagging behind the subscribers.
+ This step can be skipped if
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ has been correctly configured. If standby_slot_names is not configured
+ correctly, it is highly recommended to run this step after the primary
+ server is down, otherwise the results of the query may vary at different
+ points of time due to the ongoing replication on the logical subscribers
+ from the primary server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node check the last replayed WAL.
+ This step needs to be run on the database(s) that includes the failover
+ enabled subscription(s), to find the last replayed WAL on each database.
+<programlisting>
+test_sub=# SELECT
+ MAX(remote_lsn) AS remote_lsn_on_subscriber
+ FROM
+ ((
+ SELECT (CASE WHEN r.srsubstate = 'f' THEN pg_replication_origin_progress(CONCAT('pg_', r.srsubid, '_', r.srrelid), false)
+ WHEN r.srsubstate IN ('s', 'r') THEN r.srsublsn END) AS remote_lsn
+ FROM pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate IN ('f', 's', 'r') AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT pg_replication_origin_progress(CONCAT('pg_', s.oid), false) AS remote_lsn
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ remote_lsn_on_subscriber
+--------------------------
+ 0/3000388
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, on the standby server check that the last-received WAL location
+ is ahead of the replayed WAL location(s) on the subscriber identified
+ above. If the above SQL result was NULL, it means the subscriber has not
+ yet replayed any WAL, so the standby server must be ahead of the
+ subscriber, and this step can be skipped.
+<programlisting>
+test_standby=# SELECT pg_last_wal_receive_lsn() >= '0/3000388'::pg_lsn AS failover_ready;
+ failover_ready
+----------------
+ t
+(1 row)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+ </procedure>
+
+ <para>
+ If the result (<literal>failover_ready</literal>) of both above steps is
+ true, existing subscriptions will be able to continue without data loss.
+ </para>
+
+ </sect1>
+
<sect1 id="logical-replication-row-filter">
<title>Row Filters</title>
--
2.30.0.windows.2
[application/octet-stream] v107-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch (49.9K, ../../OS0PR01MB57161AEDBD6D3538905E069994212@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v107-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch)
download | inline diff:
From 921fb38348762cc5df4ada539e85bc583cbf8807 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Mon, 4 Mar 2024 18:01:28 +0800
Subject: [PATCH v107] Allow logical walsenders to wait for the physical
standby
This patch introduces a mechanism to ensure that physical standby servers,
which are potential failover candidates, have received and flushed changes
before making them visible to subscribers. By doing so, it guarantees that
the promoted standby server is not lagging behind the subscribers when a
failover is necessary.
A new parameter named 'standby_slot_names' is introduced. The logical
walsender now guarantees that all local changes are sent and flushed to
the standby servers corresponding to the replication slots specified in
'standby_slot_names' before sending those changes to the subscriber.
Additionally, The SQL functions pg_logical_slot_get_changes,
pg_logical_slot_peek_changes and pg_replication_slot_advance are modified.
Now, when used with logical failover slots, these functions will
block until all physical slots specified in 'standby_slot_names' have
confirmed WAL receipt.
---
doc/src/sgml/config.sgml | 45 ++
doc/src/sgml/func.sgml | 16 +-
doc/src/sgml/logicaldecoding.sgml | 12 +
.../replication/logical/logicalfuncs.c | 12 +
src/backend/replication/logical/slotsync.c | 11 +
src/backend/replication/slot.c | 408 +++++++++++++++++-
src/backend/replication/slotfuncs.c | 12 +
src/backend/replication/walsender.c | 160 ++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 14 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/replication/slot.h | 5 +
src/include/replication/walsender.h | 1 +
src/include/replication/walsender_private.h | 7 +
src/include/utils/guc_hooks.h | 3 +
src/test/recovery/t/006_logical_decoding.pl | 3 +-
.../t/040_standby_failover_slots_sync.pl | 244 ++++++++++-
src/tools/pgindent/typedefs.list | 1 +
18 files changed, 930 insertions(+), 27 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index b38cbd714a..792ab22290 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4559,6 +4559,51 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Lists the streaming replication standby server slot names that logical
+ WAL sender processes will wait for. Logical WAL sender processes will
+ send decoded changes to plugins only after the specified replication
+ slots confirm receiving WAL. This guarantees that logical replication
+ slots with failover enabled do not consume changes until those changes
+ are received and flushed to corresponding physical standbys. If a
+ logical replication connection is meant to switch to a physical standby
+ after the standby is promoted, the physical replication slot for the
+ standby should be listed here. Note that logical replication will not
+ proceed if the slots specified in the
+ <varname>standby_slot_names</varname> do not exist or are invalidated.
+ Additionally, the replication management functions
+ <link linkend="pg-replication-slot-advance">
+ <function>pg_replication_slot_advance</function></link>,
+ <link linkend="pg-logical-slot-get-changes">
+ <function>pg_logical_slot_get_changes</function></link>, and
+ <link linkend="pg-logical-slot-peek-changes">
+ <function>pg_logical_slot_peek_changes</function></link>,
+ when used with logical failover slots, will block until all
+ physical slots specified in <varname>standby_slot_names</varname> have
+ confirmed WAL receipt.
+ </para>
+ <para>
+ The standbys corresponding to the physical replication slots in
+ <varname>standby_slot_names</varname> must configure
+ <literal>sync_replication_slots = true</literal> so they can receive
+ logical failover slot changes from the primary.
+ </para>
+ <note>
+ <para>
+ The slots must be named explicitly. For example, specifying wildcard
+ values like <literal>*</literal> is not permitted.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index e5fa82c161..0bb7aeb40e 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -28150,7 +28150,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-get-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_get_changes</primary>
</indexterm>
@@ -28173,11 +28173,15 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
the specified value. Note, however, that the actual number of
rows returned may be larger, since this limit is only checked after
adding the rows produced when decoding each new transaction commit.
+ If the specified slot is a logical failover slot then the function will
+ not return until all physical slots specified in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ have confirmed WAL receipt.
</para></entry>
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-peek-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_peek_changes</primary>
</indexterm>
@@ -28232,7 +28236,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-replication-slot-advance" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_replication_slot_advance</primary>
</indexterm>
@@ -28248,7 +28252,11 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
the name of the slot and the actual position that it was advanced to.
The updated slot position information is written out at the next
checkpoint if any advancing is done. So in the event of a crash, the
- slot may return to an earlier position.
+ slot may return to an earlier position. If the specified slot is a
+ logical failover slot then the function will not return until all
+ physical slots specified in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ have confirmed WAL receipt.
</para></entry>
</row>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 930c0fa8a6..5da3ac7f4c 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -384,6 +384,18 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU
must be enabled on the standby. It is also necessary to specify a valid
<literal>dbname</literal> in the
<link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>.
+ It's highly recommended that the said physical replication slot is named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ list on the primary, to prevent the subscriber from consuming changes
+ faster than the hot standby. Even when correctly configured, some latency
+ is expected when sending changes to logical subscribers due to the waiting
+ on slots named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>.
+ When <varname>standby_slot_names</varname> is utilized, the
+ primary server will not completely shut down until the corresponding
+ standbys, associated with the physical replication slots specified
+ in <varname>standby_slot_names</varname>, have confirmed
+ receiving the WAL up to the latest flushed position on the primary server.
</para>
<para>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index a1ff631e5e..b4dd5cce75 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -105,6 +105,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
MemoryContext per_query_ctx;
MemoryContext oldcontext;
XLogRecPtr end_of_wal;
+ XLogRecPtr wait_for_wal_lsn;
LogicalDecodingContext *ctx;
ResourceOwner old_resowner = CurrentResourceOwner;
ArrayType *arr;
@@ -224,6 +225,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
NameStr(MyReplicationSlot->data.plugin),
format_procedure(fcinfo->flinfo->fn_oid))));
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to wait_for_wal_lsn.
+ */
+ if (XLogRecPtrIsInvalid(upto_lsn))
+ wait_for_wal_lsn = end_of_wal;
+ else
+ wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
+
+ WaitForStandbyConfirmation(wait_for_wal_lsn);
+
ctx->output_writer_private = p;
/*
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index ad0fc6a04b..5074c8409f 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -488,6 +488,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
latestFlushPtr = GetStandbyFlushRecPtr(NULL);
if (remote_slot->confirmed_lsn > latestFlushPtr)
{
+ /*
+ * Can get here only if GUC 'standby_slot_names' on the primary server
+ * was not configured correctly.
+ */
ereport(AmLogicalSlotSyncWorkerProcess() ? LOG : ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("skipping slot synchronization as the received slot sync"
@@ -857,6 +861,13 @@ validate_remote_info(WalReceiverConn *wrconn)
remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull));
Assert(!isnull);
+ /*
+ * Slot sync is currently not supported on a cascading standby. This is
+ * because if we allow it, the primary server needs to wait for all the
+ * cascading standbys, otherwise, logical subscribers can still be ahead
+ * of one of the cascading standbys which we plan to promote. Thus, to
+ * avoid this additional complexity, we restrict it for the time being.
+ */
if (remote_in_recovery)
ereport(ERROR,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 2614f98ddd..f80230fe40 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -46,13 +46,18 @@
#include "common/string.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/interrupt.h"
#include "replication/slotsync.h"
#include "replication/slot.h"
+#include "replication/walsender_private.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
+#include "utils/memutils.h"
+#include "utils/varlena.h"
/*
* Replication slot on-disk data structure.
@@ -77,6 +82,21 @@ typedef struct ReplicationSlotOnDisk
ReplicationSlotPersistentData slotdata;
} ReplicationSlotOnDisk;
+/*
+ * Struct for the configuration of standby_slot_names.
+ *
+ * Note: this must be a flat representation that can be held in a single chunk
+ * of guc_malloc'd memory, so that it can be stored as the "extra" data for the
+ * standby_slot_names GUC.
+ */
+typedef struct
+{
+ int slot_num;
+
+ /* slot_names contains nmembers consecutive nul-terminated C strings */
+ char slot_names[FLEXIBLE_ARRAY_MEMBER];
+} StandbySlotConfigData;
+
/*
* Lookup table for slot invalidation causes.
*/
@@ -115,10 +135,25 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
/* My backend's replication slot in the shared memory array */
ReplicationSlot *MyReplicationSlot = NULL;
-/* GUC variable */
+/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+/*
+ * This GUC lists streaming replication standby server slot names that
+ * logical WAL sender processes will wait for.
+ */
+char *standby_slot_names;
+
+/* This is parsed and cached configuration for standby_slot_names */
+static StandbySlotConfigData *standby_slot_config;
+
+/*
+ * Oldest LSN that has been confirmed to be flushed to the standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
+
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
@@ -2345,3 +2380,374 @@ GetSlotInvalidationCause(const char *conflict_reason)
Assert(found);
return result;
}
+
+/*
+ * A helper function to validate slots specified in GUC standby_slot_names.
+ *
+ * The rawname will be parsed, and the parsed result will be saved into
+ * *elemlist.
+ */
+static bool
+validate_standby_slots(char *rawname, List **elemlist)
+{
+ bool ok;
+
+ /* Verify syntax and parse string into a list of identifiers */
+ ok = SplitIdentifierString(rawname, ',', elemlist);
+
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
+ else if (!ReplicationSlotCtl)
+ {
+ /*
+ * We cannot validate the replication slot if the replication slots'
+ * data has not been initialized. This is ok as we will anyway
+ * validate the specified slot when waiting for them to catch up. See
+ * StandbySlotsHaveCaughtup() for details.
+ */
+ }
+ else
+ {
+ /* Check that the specified slots exist and are logical slots */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, *elemlist)
+ {
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ GUC_check_errdetail("replication slot \"%s\" does not exist",
+ name);
+ ok = false;
+ break;
+ }
+
+ if (!SlotIsPhysical(slot))
+ {
+ GUC_check_errdetail("\"%s\" is not a physical replication slot",
+ name);
+ ok = false;
+ break;
+ }
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+ }
+
+ return ok;
+}
+
+/*
+ * GUC check_hook for standby_slot_names
+ */
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ char *rawname;
+ char *ptr;
+ List *elemlist;
+ int size;
+ StandbySlotConfigData *config;
+
+ if ((*newval)[0] == '\0')
+ return true;
+
+ /*
+ * "*" is not accepted as in that case the primary will not be able to
+ * know for which all standbys to wait for. It is inappropriate to block
+ * logical replication for physical slots that either lack associated
+ * standbys or have standbys associated that are not enabled for
+ * replication slot synchronization.
+ */
+ if (strcmp(*newval, "*") == 0)
+ {
+ GUC_check_errdetail("\"*\" is not accepted for standby_slot_names");
+ return false;
+ }
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Now verify if the specified slots exist and have correct type */
+ if (!validate_standby_slots(rawname, &elemlist))
+ {
+ pfree(rawname);
+ list_free(elemlist);
+ return false;
+ }
+
+ /* Compute the size required for the StandbySlotConfigData struct */
+ size = offsetof(StandbySlotConfigData, slot_names);
+ foreach_ptr(char, slot_name, elemlist)
+ size += strlen(slot_name) + 1;
+
+ /* GUC extra value must be guc_malloc'd, not palloc'd */
+ config = (StandbySlotConfigData *) guc_malloc(LOG, size);
+
+ /* Transform the data into StandbySlotConfigData */
+ config->slot_num = list_length(elemlist);
+
+ ptr = config->slot_names;
+ foreach_ptr(char, slot_name, elemlist)
+ {
+ strcpy(ptr, slot_name);
+ ptr += strlen(slot_name) + 1;
+ }
+
+ *extra = (void *) config;
+
+ pfree(rawname);
+ list_free(elemlist);
+ return true;
+}
+
+/*
+ * GUC assign_hook for standby_slot_names
+ */
+void
+assign_standby_slot_names(const char *newval, void *extra)
+{
+ /*
+ * The standby slots may have changed, so we must recompute the oldest
+ * LSN.
+ */
+ ss_oldest_flush_lsn = InvalidXLogRecPtr;
+
+ standby_slot_config = (StandbySlotConfigData *) extra;
+}
+
+/*
+ * Return true if the currently acquired slot is specified in
+ * standby_slot_names GUC; otherwise, return false.
+ */
+bool
+AcquiredStandbySlot(void)
+{
+ const char *name;
+
+ /* Return false if there is no value in standby_slot_names */
+ if (standby_slot_config == NULL)
+ return false;
+
+ name = standby_slot_config->slot_names;
+ for (int i = 0; i < standby_slot_config->slot_num; i++)
+ {
+ if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0)
+ return true;
+
+ name += strlen(name) + 1;
+ }
+
+ return false;
+}
+
+/*
+ * Return true if the slots specified in standby_slot_names have caught up to
+ * the given WAL location, false otherwise.
+ *
+ * The elevel parameter specifies the error level used for logging messages
+ * related to slots that do not exist, are invalidated, or are inactive.
+ */
+bool
+StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel)
+{
+ const char *name;
+ int caught_up_slot_num = 0;
+ XLogRecPtr min_restart_lsn = InvalidXLogRecPtr;
+
+ /*
+ * Don't need to wait for the standbys to catch up if there is no value in
+ * standby_slot_names.
+ */
+ if (standby_slot_config == NULL)
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if we are on a standby
+ * server, since we do not support syncing slots to cascading standbys.
+ */
+ if (RecoveryInProgress())
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if they are already
+ * beyond the specified WAL location.
+ */
+ if (!XLogRecPtrIsInvalid(ss_oldest_flush_lsn) &&
+ ss_oldest_flush_lsn >= wait_for_lsn)
+ return true;
+
+ /*
+ * To prevent concurrent slot dropping and creation while filtering the
+ * slots, take the ReplicationSlotControlLock outside of the loop.
+ */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ name = standby_slot_config->slot_names;
+ for (int i = 0; i < standby_slot_config->slot_num; i++)
+ {
+ XLogRecPtr restart_lsn;
+ bool invalidated;
+ bool inactive;
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ /*
+ * If a slot name provided in standby_slot_names does not exist,
+ * report a message and exit the loop. A user can specify a slot
+ * name that does not exist just before the server startup. The
+ * GUC check_hook(validate_standby_slots) cannot validate such a
+ * slot during startup as the ReplicationSlotCtl shared memory is
+ * not initialized at that time. It is also possible for a user to
+ * drop the slot in standby_slot_names afterwards.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not exist",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider creating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (SlotIsLogical(slot))
+ {
+ /*
+ * If a logical slot name is provided in standby_slot_names,
+ * report a message and exit the loop. Similar to the non-existent
+ * case, a user can specify a logical slot name in
+ * standby_slot_names before the server startup, or drop an
+ * existing physical slot and recreate a logical slot with the
+ * same name.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot have logical replication slot \"%s\" in parameter %s",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting for correction on \"%s\".",
+ name),
+ errhint("Consider removing logical slot \"%s\" from parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ SpinLockAcquire(&slot->mutex);
+ restart_lsn = slot->data.restart_lsn;
+ invalidated = slot->data.invalidated != RS_INVAL_NONE;
+ inactive = slot->active_pid == 0;
+ SpinLockRelease(&slot->mutex);
+
+ if (invalidated)
+ {
+ /* Specified physical slot has been invalidated */
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("physical slot \"%s\" specified in parameter %s has been invalidated",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider dropping and recreating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (XLogRecPtrIsInvalid(restart_lsn) || restart_lsn < wait_for_lsn)
+ {
+ /* Log a message if no active_pid for this physical slot */
+ if (inactive)
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not have active_pid",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider starting standby associated with \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+
+ /* Continue if the current slot hasn't caught up. */
+ break;
+ }
+
+ Assert(restart_lsn >= wait_for_lsn);
+
+ if (XLogRecPtrIsInvalid(min_restart_lsn) ||
+ min_restart_lsn > restart_lsn)
+ min_restart_lsn = restart_lsn;
+
+ caught_up_slot_num++;
+
+ name += strlen(name) + 1;
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /*
+ * Return false if not all the standbys have caught up to the specified
+ * WAL location.
+ */
+ if (caught_up_slot_num != standby_slot_config->slot_num)
+ return false;
+
+ /* The ss_oldest_flush_lsn must not retreat. */
+ Assert(XLogRecPtrIsInvalid(ss_oldest_flush_lsn) ||
+ min_restart_lsn >= ss_oldest_flush_lsn);
+
+ ss_oldest_flush_lsn = min_restart_lsn;
+
+ return true;
+}
+
+/*
+ * Wait for physical standbys to confirm receiving the given lsn.
+ *
+ * Used by logical decoding SQL functions. It waits for physical standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+void
+WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
+{
+ /*
+ * Don't need to wait for the standby to catch up if the current acquired
+ * slot is not a logical failover slot, or there is no value in
+ * standby_slot_names.
+ */
+ if (!MyReplicationSlot->data.failover || !standby_slot_config)
+ return;
+
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+
+ for (;;)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /* Exit if done waiting for every slot. */
+ if (StandbySlotsHaveCaughtup(wait_for_lsn, WARNING))
+ break;
+
+ /*
+ * Wait for the slots in the standby_slot_names to catch up, but use a
+ * timeout (1s) so we can also check if the standby_slot_names has
+ * been changed.
+ */
+ ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000,
+ WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION);
+ }
+
+ ConditionVariableCancelSleep();
+}
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 768a304723..ad79e1fccd 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -464,6 +464,12 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* crash, but this makes the data consistent after a clean shutdown.
*/
ReplicationSlotMarkDirty();
+
+ /*
+ * Wake up logical walsenders holding logical failover slots after
+ * updating the restart_lsn of the physical slot.
+ */
+ PhysicalWakeupLogicalWalSnd();
}
return retlsn;
@@ -504,6 +510,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto)
.segment_close = wal_segment_close),
NULL, NULL, NULL);
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to moveto lsn.
+ */
+ WaitForStandbyConfirmation(moveto);
+
/*
* Start reading at the slot's restart_lsn, which we know to point to
* a valid record.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 0f1047179c..d8bdc65dc1 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1725,26 +1725,111 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
ProcessPendingWrites();
}
+/*
+ * Wake up the logical walsender processes with logical failover slots if the
+ * currently acquired physical slot is specified in standby_slot_names GUC.
+ */
+void
+PhysicalWakeupLogicalWalSnd(void)
+{
+ Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot));
+
+ /*
+ * If we are running in a standby, there is no need to wake up walsenders.
+ * This is because we do not support syncing slots to cascading standbys,
+ * so, there are no walsenders waiting for standbys to catch up.
+ */
+ if (RecoveryInProgress())
+ return;
+
+ if (AcquiredStandbySlot())
+ ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv);
+}
+
+/*
+ * Returns true if not all standbys have caught up to the flushed position
+ * (flushed_lsn) when the current acquired slot is a logical failover
+ * slot and we are streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForStandbys(XLogRecPtr flushed_lsn, uint32 *wait_event)
+{
+ int elevel = got_STOPPING ? ERROR : WARNING;
+ bool failover_slot;
+
+ failover_slot = (replication_active && MyReplicationSlot->data.failover);
+
+ /*
+ * Note that after receiving the shutdown signal, an ERROR is reported if
+ * any slots are dropped, invalidated, or inactive. This measure is taken
+ * to prevent the walsender from waiting indefinitely.
+ */
+ if (failover_slot && !StandbySlotsHaveCaughtup(flushed_lsn, elevel))
+ {
+ *wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
+ return true;
+ }
+
+ *wait_event = 0;
+ return false;
+}
+
+/*
+ * Returns true if we need to wait for WALs to be flushed to disk, or if not
+ * all standbys have caught up to the flushed position (flushed_lsn) when the
+ * current acquired slot is a logical failover slot and we are
+ * streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
+{
+ /* Check if we need to wait for WALs to be flushed to disk */
+ if (target_lsn > flushed_lsn)
+ {
+ *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
+ return true;
+ }
+
+ /* Check if the standby slots have caught up to the flushed position */
+ return NeedToWaitForStandbys(flushed_lsn, wait_event);
+}
+
/*
* Wait till WAL < loc is flushed to disk so it can be safely sent to client.
*
- * Returns end LSN of flushed WAL. Normally this will be >= loc, but
- * if we detect a shutdown request (either from postmaster or client)
- * we will return early, so caller must always check.
+ * If the walsender holds a logical slot that has enabled failover, we also
+ * wait for all the specified streaming replication standby servers to confirm
+ * receipt of WAL up to RecentFlushPtr. It is beneficial to wait here for the
+ * confirmation up to RecentFlushPtr rather than waiting before transmitting
+ * each change to logical subscribers, which is already covered by
+ * RecentFlushPtr.
+ *
+ * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we
+ * detect a shutdown request (either from postmaster or client) we will return
+ * early, so caller must always check.
*/
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
int wakeEvents;
+ uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
- * have enough WAL available. This is particularly interesting if we're
- * far behind.
+ * have enough WAL available and all the standby servers have confirmed
+ * receipt of WAL up to RecentFlushPtr. This is particularly interesting
+ * if we're far behind.
*/
- if (RecentFlushPtr != InvalidXLogRecPtr &&
- loc <= RecentFlushPtr)
+ if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
return RecentFlushPtr;
/* Get a more recent flush pointer. */
@@ -1753,8 +1838,14 @@ WalSndWaitForWal(XLogRecPtr loc)
else
RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * Within the loop, we wait for the necessary WALs to be flushed to disk
+ * first, followed by waiting for standbys to catch up if there are enough
+ * WALs (see NeedToWaitForWal()) or upon receiving the shutdown signal.
+ */
for (;;)
{
+ bool wait_for_standby_at_stop = false;
long sleeptime;
/* Clear any already-pending wakeups */
@@ -1781,21 +1872,35 @@ WalSndWaitForWal(XLogRecPtr loc)
if (got_STOPPING)
XLogBackgroundFlush();
- /* Update our idea of the currently flushed position. */
- if (!RecoveryInProgress())
- RecentFlushPtr = GetFlushRecPtr(NULL);
- else
- RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * To avoid the scenario where standbys need to catch up to a newer
+ * WAL location in each iteration, we update our idea of the currently
+ * flushed position only if we are not waiting for standbys to catch
+ * up.
+ */
+ if (wait_event != WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ {
+ if (!RecoveryInProgress())
+ RecentFlushPtr = GetFlushRecPtr(NULL);
+ else
+ RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ }
/*
- * If postmaster asked us to stop, don't wait anymore.
+ * If postmaster asked us to stop and the standby slots have caught up
+ * to the flushed position, don't wait anymore.
*
* It's important to do this check after the recomputation of
* RecentFlushPtr, so we can send all remaining data before shutting
* down.
*/
if (got_STOPPING)
- break;
+ {
+ if (NeedToWaitForStandbys(RecentFlushPtr, &wait_event))
+ wait_for_standby_at_stop = true;
+ else
+ break;
+ }
/*
* We only send regular messages to the client for full decoded
@@ -1810,11 +1915,18 @@ WalSndWaitForWal(XLogRecPtr loc)
!waiting_for_ping_response)
WalSndKeepalive(false, InvalidXLogRecPtr);
- /* check whether we're done */
- if (loc <= RecentFlushPtr)
+ /*
+ * Exit the loop if already caught up and doesn't need to wait for
+ * standby slots.
+ */
+ if (!wait_for_standby_at_stop &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
break;
- /* Waiting for new WAL. Since we need to wait, we're now caught up. */
+ /*
+ * Waiting for new WAL or waiting for standbys to catch up. Since we
+ * need to wait, we're now caught up.
+ */
WalSndCaughtUp = true;
/*
@@ -1852,7 +1964,9 @@ WalSndWaitForWal(XLogRecPtr loc)
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL);
+ Assert(wait_event != 0);
+
+ WalSndWait(wakeEvents, sleeptime, wait_event);
}
/* reactivate latch so WalSndLoop knows to continue */
@@ -2262,6 +2376,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn)
{
ReplicationSlotMarkDirty();
ReplicationSlotsComputeRequiredLSN();
+ PhysicalWakeupLogicalWalSnd();
}
/*
@@ -3535,6 +3650,7 @@ WalSndShmemInit(void)
ConditionVariableInit(&WalSndCtl->wal_flush_cv);
ConditionVariableInit(&WalSndCtl->wal_replay_cv);
+ ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv);
}
}
@@ -3604,8 +3720,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event)
*
* And, we use separate shared memory CVs for physical and logical
* walsenders for selective wake ups, see WalSndWakeup() for more details.
+ *
+ * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV
+ * until awakened by physical walsenders after the walreceiver confirms
+ * the receipt of the LSN.
*/
- if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
+ if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+ else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv);
else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index ec2f31f82a..c08e00d1d6 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP
LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server."
LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server."
SSL_OPEN_SERVER "Waiting for SSL while attempting connection."
+WAIT_FOR_STANDBY_CONFIRMATION "Waiting for WAL to be received and flushed by the physical standby."
WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process."
WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 45013582a7..d77214795d 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4670,6 +4670,20 @@ struct config_string ConfigureNamesString[] =
check_debug_io_direct, assign_debug_io_direct, NULL
},
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("Lists streaming replication standby server slot "
+ "names that logical WAL sender processes will wait for."),
+ gettext_noop("Logical WAL sender processes will send decoded "
+ "changes to plugins only after the specified "
+ "replication slots confirm receiving WAL."),
+ GUC_LIST_INPUT
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, assign_standby_slot_names, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index edcc0282b2..2244ee52f7 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -343,6 +343,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsender processes will wait for
# - Standby Servers -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index acbf567150..73f7154173 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -226,6 +226,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
+extern PGDLLIMPORT char *standby_slot_names;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -274,4 +275,8 @@ extern void CheckSlotPermissions(void);
extern ReplicationSlotInvalidationCause
GetSlotInvalidationCause(const char *conflict_reason);
+extern bool AcquiredStandbySlot(void);
+extern bool StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel);
+extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn);
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index 0c3996e926..f2d8297f01 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -39,6 +39,7 @@ extern void InitWalSender(void);
extern bool exec_replication_command(const char *cmd_string);
extern void WalSndErrorCleanup(void);
extern void WalSndResourceCleanup(bool isCommit);
+extern void PhysicalWakeupLogicalWalSnd(void);
extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
extern void WalSndSignals(void);
extern Size WalSndShmemSize(void);
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 3113e9ea47..109924ffcd 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -113,6 +113,13 @@ typedef struct
ConditionVariable wal_flush_cv;
ConditionVariable wal_replay_cv;
+ /*
+ * Used by physical walsenders holding slots specified in
+ * standby_slot_names to wake up logical walsenders holding logical
+ * failover slots when a walreceiver confirms the receipt of LSN.
+ */
+ ConditionVariable wal_confirm_rcv_cv;
+
WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER];
} WalSndCtlData;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index c8a7aa9a11..d64dc5fcdb 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra,
extern void assign_wal_consistency_checking(const char *newval, void *extra);
extern bool check_wal_segment_size(int *newval, void **extra, GucSource source);
extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
+extern bool check_standby_slot_names(char **newval, void **extra,
+ GucSource source);
+extern void assign_standby_slot_names(const char *newval, void *extra);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl
index 5c7b4ca5e3..b95d95c06f 100644
--- a/src/test/recovery/t/006_logical_decoding.pl
+++ b/src/test/recovery/t/006_logical_decoding.pl
@@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'},
undef, 'logical slot was actually dropped with DB');
# Test logical slot advancing and its durability.
+# Passing failover=true (last arg) should not have any impact on advancing.
my $logical_slot = 'logical_slot';
$node_primary->safe_psql('postgres',
- "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);"
+ "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);"
);
$node_primary->psql(
'postgres', "
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 021c58f621..99025b352a 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -504,18 +504,258 @@ ok( $standby1->poll_query_until(
"SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot' AND synced AND NOT temporary;"),
'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby');
+##################################################
+# Test that logical failover replication slots wait for the specified
+# physical replication slots to receive the changes first. It uses the
+# following set up:
+#
+# (physical standbys)
+# | ----> standby1 (primary_slot_name = sb1_slot)
+# | ----> standby2 (primary_slot_name = sb2_slot)
+# primary ----- |
+# (logical replication)
+# | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
+# | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
+#
+# standby_slot_names = 'sb1_slot'
+#
+# The setup is configured in such a way that the logical slot of subscriber1 is
+# enabled for failover, and thus the subscriber1 will wait for the physical
+# slot of standby1(sb1_slot) to catch up before receiving the decoded changes.
+##################################################
+
+$backup_name = 'backup3';
+
+$primary->psql('postgres',
+ q{SELECT pg_create_physical_replication_slot('sb2_slot');});
+
+$primary->backup($backup_name);
+
+# Create another standby
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup(
+ $primary, $backup_name,
+ has_streaming => 1,
+ has_restoring => 1);
+$standby2->append_conf(
+ 'postgresql.conf', qq(
+primary_slot_name = 'sb2_slot'
+));
+$standby2->start;
+$primary->wait_for_replay_catchup($standby2);
+
+# Configure primary to disallow any logical slots that have enabled failover
+# from getting ahead of the specified physical replication slot (sb1_slot).
+$primary->append_conf(
+ 'postgresql.conf', qq(
+standby_slot_names = 'sb1_slot'
+));
+$primary->reload;
+
+# Create another subscriber node without enabling failover, wait for sync to
+# complete
+my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$subscriber2->init;
+$subscriber2->start;
+$subscriber2->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE tab_int (a int PRIMARY KEY);
+ CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot);
+]);
+
+$subscriber2->wait_for_subscription_sync;
+
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+my $offset = -s $primary->logfile;
+
+# Stop the standby associated with the specified physical replication slot
+# (sb1_slot) so that the logical replication slot (lsub1_slot) won't receive
+# changes until the standby comes up.
+$standby1->stop;
+
+# Create some data on the primary
+my $primary_row_count = 20;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(11, $primary_row_count);");
+
+# Wait until the standby2 that's still running gets the data from the primary
+$primary->wait_for_replay_catchup($standby2);
+$result = $standby2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby2 gets data from primary");
+
+# Wait for regress_mysub2 to get the data from the primary. This subscription
+# was not enabled for failover so it gets the data without waiting for any
+# standbys.
+$primary->wait_for_catchup('regress_mysub2');
+$result = $subscriber2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "subscriber2 gets data from primary");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# The regress_mysub1 was enabled for failover so it doesn't get the data from
+# primary and keeps waiting for the standby specified in standby_slot_names
+# (sb1_slot aka standby1).
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) <> $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data from primary until standby1 acknowledges changes"
+);
+
+# Start the standby specified in standby_slot_names (sb1_slot aka standby1) and
+# wait for it to catch up with the primary.
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+$result = $standby1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby1 gets data from primary");
+
+# Now that the standby specified in standby_slot_names is up and running, the
+# primary can send the decoded changes to the subscription enabled for failover
+# (i.e. regress_mysub1). While the standby was down, regress_mysub1 didn't
+# receive any data from the primary. i.e. the primary didn't allow it to go
+# ahead of standby.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 acknowledges changes");
+
+##################################################
+# Verify that when using pg_logical_slot_get_changes to consume changes from a
+# logical slot with failover enabled, it will also wait for the slots specified
+# in standby_slot_names to catch up.
+##################################################
+
+# Stop the standby associated with the specified physical replication slot so
+# that the logical replication slot won't receive changes until the standby
+# slot's restart_lsn is advanced or the slot is removed from the
+# standby_slot_names list.
+$primary->safe_psql('postgres', "TRUNCATE tab_int;");
+$primary->wait_for_catchup('regress_mysub1');
+$standby1->stop;
+
+# Disable the regress_mysub1 to prevent the logical walsender from generating
+# more warnings.
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 DISABLE");
+
+# Wait for the replication slot to become inactive on the publisher
+$primary->poll_query_until(
+ 'postgres',
+ "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'",
+ 1);
+
+# Create a logical 'test_decoding' replication slot with failover enabled
+$primary->safe_psql('postgres',
+ "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);"
+);
+
+my $back_q = $primary->background_psql(
+ 'postgres',
+ on_error_stop => 0,
+ timeout => $PostgreSQL::Test::Utils::timeout_default);
+
+# pg_logical_slot_get_changes will be blocked until the standby catches up,
+# hence it needs to be executed in a background session.
+$offset = -s $primary->logfile;
+$back_q->query_until(
+ qr/logical_slot_get_changes/, q(
+ \echo logical_slot_get_changes
+ SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);
+));
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the function
+# pg_logical_slot_get_changes should now return, and the session can be
+# stopped.
+$back_q->quit;
+
+$primary->safe_psql('postgres',
+ "SELECT pg_drop_replication_slot('test_slot');"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
+# Enable the regress_mysub1 for further tests
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+##################################################
+# Test that logical replication will wait for the user-created inactive
+# physical slot to catch up until we remove the slot from standby_slot_names.
+##################################################
+
+$offset = -s $primary->logfile;
+
+# Create some data on the primary
+$primary_row_count = 10;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# The regress_mysub1 doesn't get the data from primary because the specified
+# standby slot (sb1_slot) in standby_slot_names is inactive.
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data as the sb1_slot doesn't catch up");
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the primary server should now
+# send the decoded changes to the subscription.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
##################################################
# Promote the standby1 to primary. Confirm that:
# a) the slot 'lsub1_slot' is retained on the new primary
# b) logical replication for regress_mysub1 is resumed successfully after failover
##################################################
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+
$standby1->promote;
# Update subscription with the new primary's connection info
my $standby1_conninfo = $standby1->connstr . ' dbname=postgres';
$subscriber1->safe_psql('postgres',
- "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo';
- ALTER SUBSCRIPTION regress_mysub1 ENABLE; ");
+ "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo';");
# Confirm the synced slot 'lsub1_slot' is retained on the new primary
is($standby1->safe_psql('postgres',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 95ae7845d8..c0b9875838 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2650,6 +2650,7 @@ SplitPoint
SplitTextOutputData
SplitVar
StackElem
+StandbySlotConfigData
StartDataPtrType
StartLOPtrType
StartLOsPtrType
--
2.30.0.windows.2
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-06 13:24 Zhijie Hou (Fujitsu) <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 2 replies; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-06 13:24 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Wednesday, March 6, 2024 9:13 PM Zhijie Hou (Fujitsu) <[email protected]> wrote:
>
> On Wednesday, March 6, 2024 11:04 AM Zhijie Hou (Fujitsu)
> <[email protected]> wrote:
> >
> > On Wednesday, March 6, 2024 9:30 AM Masahiko Sawada
> > <[email protected]> wrote:
> >
> > Hi,
> >
> > > On Fri, Mar 1, 2024 at 4:21 PM Zhijie Hou (Fujitsu)
> > > <[email protected]>
> > > wrote:
> > > >
> > > > On Friday, March 1, 2024 2:11 PM Masahiko Sawada
> > > <[email protected]> wrote:
> > > > >
> > > > >
> > > > > ---
> > > > > +void
> > > > > +assign_standby_slot_names(const char *newval, void *extra) {
> > > > > + List *standby_slots;
> > > > > + MemoryContext oldcxt;
> > > > > + char *standby_slot_names_cpy = extra;
> > > > > +
> > > > >
> > > > > Given that the newval and extra have the same data
> > > > > (standby_slot_names value), why do we not use newval instead? I
> > > > > think that if we use newval, we don't need to guc_strdup() in
> > > > > check_standby_slot_names(), we might need to do list_copy_deep()
> > > > > instead, though. It's not clear to me as there is no comment.
> > > >
> > > > I think SplitIdentifierString will modify the passed in string, so
> > > > we'd better not pass the newval to it, otherwise the stored guc
> > > > string(standby_slot_names) will be changed. I can see we are doing
> > > > similar thing in other GUC check/assign function as well.
> > > > (check_wal_consistency_checking/ assign_wal_consistency_checking,
> > > > check_createrole_self_grant/ assign_createrole_self_grant ...).
> > >
> > > Why does it have to be a List in the first place?
> >
> > I thought the List type is convenient to use here, as we have existing
> > list build function(SplitIdentifierString), and have convenient list
> > macro to loop the
> > list(foreach_ptr) which can save some codes.
> >
> > > In earlier version patches, we
> > > used to copy the list and delete the element until it became empty,
> > > while waiting for physical wal senders. But we now just refer to
> > > each slot name in the list. The current code assumes that
> > > stnadby_slot_names_cpy is allocated in GUCMemoryContext but once it
> > > changes, it will silently get broken. I think we can check and
> > > assign standby_slot_names in a similar way to
> > > check/assign_temp_tablespaces and
> check/assign_synchronous_standby_names.
> >
> > Yes, we could do follow it by allocating an array and copy each slot
> > name into it, but it also requires some codes to build and scan the
> > array. So, is it possible to expose the GucMemorycontext or have an API like
> guc_copy_list instead ?
> > If we don't want to touch the guc api, I am ok with using an array as well.
>
> I rethink about this and realize that it's not good to do the memory allocation in
> assign hook function. As the "src/backend/utils/misc/README" said, we'd
> better do that in check hook function and pass it via extra to assign hook
> function. And thus array is a good choice in this case rather than a List which
> cannot be passed to *extra.
>
> Here is the V107 patch set which parse and cache the standby slot names in an
> array instead of a List.
The patch needs to be rebased due to recent commit.
Attach the V107_2 path set. There are no code changes in this version.
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v107_2-0001-Allow-logical-walsenders-to-wait-for-the-physi.patch (49.9K, ../../OS0PR01MB57161D93206707DAF900C34F94212@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v107_2-0001-Allow-logical-walsenders-to-wait-for-the-physi.patch)
download | inline diff:
From 8e758032ddde2ddcb3983b8738245d2aeb610018 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Wed, 6 Mar 2024 21:15:43 +0800
Subject: [PATCH v1072] Allow logical walsenders to wait for the physical
standby
This patch introduces a mechanism to ensure that physical standby servers,
which are potential failover candidates, have received and flushed changes
before making them visible to subscribers. By doing so, it guarantees that
the promoted standby server is not lagging behind the subscribers when a
failover is necessary.
A new parameter named 'standby_slot_names' is introduced. The logical
walsender now guarantees that all local changes are sent and flushed to
the standby servers corresponding to the replication slots specified in
'standby_slot_names' before sending those changes to the subscriber.
Additionally, The SQL functions pg_logical_slot_get_changes,
pg_logical_slot_peek_changes and pg_replication_slot_advance are modified.
Now, when used with logical failover slots, these functions will
block until all physical slots specified in 'standby_slot_names' have
confirmed WAL receipt.
---
doc/src/sgml/config.sgml | 45 ++
doc/src/sgml/func.sgml | 16 +-
doc/src/sgml/logicaldecoding.sgml | 12 +
.../replication/logical/logicalfuncs.c | 12 +
src/backend/replication/logical/slotsync.c | 11 +
src/backend/replication/slot.c | 407 +++++++++++++++++-
src/backend/replication/slotfuncs.c | 12 +
src/backend/replication/walsender.c | 160 ++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 14 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/replication/slot.h | 5 +
src/include/replication/walsender.h | 1 +
src/include/replication/walsender_private.h | 7 +
src/include/utils/guc_hooks.h | 3 +
src/test/recovery/t/006_logical_decoding.pl | 3 +-
.../t/040_standby_failover_slots_sync.pl | 244 ++++++++++-
src/tools/pgindent/typedefs.list | 1 +
18 files changed, 929 insertions(+), 27 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index b38cbd714a..792ab22290 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4559,6 +4559,51 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Lists the streaming replication standby server slot names that logical
+ WAL sender processes will wait for. Logical WAL sender processes will
+ send decoded changes to plugins only after the specified replication
+ slots confirm receiving WAL. This guarantees that logical replication
+ slots with failover enabled do not consume changes until those changes
+ are received and flushed to corresponding physical standbys. If a
+ logical replication connection is meant to switch to a physical standby
+ after the standby is promoted, the physical replication slot for the
+ standby should be listed here. Note that logical replication will not
+ proceed if the slots specified in the
+ <varname>standby_slot_names</varname> do not exist or are invalidated.
+ Additionally, the replication management functions
+ <link linkend="pg-replication-slot-advance">
+ <function>pg_replication_slot_advance</function></link>,
+ <link linkend="pg-logical-slot-get-changes">
+ <function>pg_logical_slot_get_changes</function></link>, and
+ <link linkend="pg-logical-slot-peek-changes">
+ <function>pg_logical_slot_peek_changes</function></link>,
+ when used with logical failover slots, will block until all
+ physical slots specified in <varname>standby_slot_names</varname> have
+ confirmed WAL receipt.
+ </para>
+ <para>
+ The standbys corresponding to the physical replication slots in
+ <varname>standby_slot_names</varname> must configure
+ <literal>sync_replication_slots = true</literal> so they can receive
+ logical failover slot changes from the primary.
+ </para>
+ <note>
+ <para>
+ The slots must be named explicitly. For example, specifying wildcard
+ values like <literal>*</literal> is not permitted.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index e5fa82c161..0bb7aeb40e 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -28150,7 +28150,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-get-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_get_changes</primary>
</indexterm>
@@ -28173,11 +28173,15 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
the specified value. Note, however, that the actual number of
rows returned may be larger, since this limit is only checked after
adding the rows produced when decoding each new transaction commit.
+ If the specified slot is a logical failover slot then the function will
+ not return until all physical slots specified in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ have confirmed WAL receipt.
</para></entry>
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-peek-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_peek_changes</primary>
</indexterm>
@@ -28232,7 +28236,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-replication-slot-advance" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_replication_slot_advance</primary>
</indexterm>
@@ -28248,7 +28252,11 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
the name of the slot and the actual position that it was advanced to.
The updated slot position information is written out at the next
checkpoint if any advancing is done. So in the event of a crash, the
- slot may return to an earlier position.
+ slot may return to an earlier position. If the specified slot is a
+ logical failover slot then the function will not return until all
+ physical slots specified in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ have confirmed WAL receipt.
</para></entry>
</row>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8f13780e74..5a15bbc580 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -384,6 +384,18 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU
must be enabled on the standby. It is also necessary to specify a valid
<literal>dbname</literal> in the
<link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>.
+ It's highly recommended that the said physical replication slot is named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ list on the primary, to prevent the subscriber from consuming changes
+ faster than the hot standby. Even when correctly configured, some latency
+ is expected when sending changes to logical subscribers due to the waiting
+ on slots named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>.
+ When <varname>standby_slot_names</varname> is utilized, the
+ primary server will not completely shut down until the corresponding
+ standbys, associated with the physical replication slots specified
+ in <varname>standby_slot_names</varname>, have confirmed
+ receiving the WAL up to the latest flushed position on the primary server.
</para>
<para>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index a1ff631e5e..b4dd5cce75 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -105,6 +105,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
MemoryContext per_query_ctx;
MemoryContext oldcontext;
XLogRecPtr end_of_wal;
+ XLogRecPtr wait_for_wal_lsn;
LogicalDecodingContext *ctx;
ResourceOwner old_resowner = CurrentResourceOwner;
ArrayType *arr;
@@ -224,6 +225,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
NameStr(MyReplicationSlot->data.plugin),
format_procedure(fcinfo->flinfo->fn_oid))));
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to wait_for_wal_lsn.
+ */
+ if (XLogRecPtrIsInvalid(upto_lsn))
+ wait_for_wal_lsn = end_of_wal;
+ else
+ wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
+
+ WaitForStandbyConfirmation(wait_for_wal_lsn);
+
ctx->output_writer_private = p;
/*
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index ad0fc6a04b..5074c8409f 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -488,6 +488,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
latestFlushPtr = GetStandbyFlushRecPtr(NULL);
if (remote_slot->confirmed_lsn > latestFlushPtr)
{
+ /*
+ * Can get here only if GUC 'standby_slot_names' on the primary server
+ * was not configured correctly.
+ */
ereport(AmLogicalSlotSyncWorkerProcess() ? LOG : ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("skipping slot synchronization as the received slot sync"
@@ -857,6 +861,13 @@ validate_remote_info(WalReceiverConn *wrconn)
remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull));
Assert(!isnull);
+ /*
+ * Slot sync is currently not supported on a cascading standby. This is
+ * because if we allow it, the primary server needs to wait for all the
+ * cascading standbys, otherwise, logical subscribers can still be ahead
+ * of one of the cascading standbys which we plan to promote. Thus, to
+ * avoid this additional complexity, we restrict it for the time being.
+ */
if (remote_in_recovery)
ereport(ERROR,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 02ae27499b..46959ffa42 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -46,14 +46,18 @@
#include "common/string.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/interrupt.h"
#include "replication/slotsync.h"
#include "replication/slot.h"
+#include "replication/walsender_private.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
#include "utils/injection_point.h"
+#include "utils/varlena.h"
/*
* Replication slot on-disk data structure.
@@ -78,6 +82,21 @@ typedef struct ReplicationSlotOnDisk
ReplicationSlotPersistentData slotdata;
} ReplicationSlotOnDisk;
+/*
+ * Struct for the configuration of standby_slot_names.
+ *
+ * Note: this must be a flat representation that can be held in a single chunk
+ * of guc_malloc'd memory, so that it can be stored as the "extra" data for the
+ * standby_slot_names GUC.
+ */
+typedef struct
+{
+ int slot_num;
+
+ /* slot_names contains nmembers consecutive nul-terminated C strings */
+ char slot_names[FLEXIBLE_ARRAY_MEMBER];
+} StandbySlotConfigData;
+
/*
* Lookup table for slot invalidation causes.
*/
@@ -116,10 +135,25 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
/* My backend's replication slot in the shared memory array */
ReplicationSlot *MyReplicationSlot = NULL;
-/* GUC variable */
+/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+/*
+ * This GUC lists streaming replication standby server slot names that
+ * logical WAL sender processes will wait for.
+ */
+char *standby_slot_names;
+
+/* This is parsed and cached configuration for standby_slot_names */
+static StandbySlotConfigData *standby_slot_config;
+
+/*
+ * Oldest LSN that has been confirmed to be flushed to the standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
+
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
@@ -2354,3 +2388,374 @@ GetSlotInvalidationCause(const char *conflict_reason)
Assert(found);
return result;
}
+
+/*
+ * A helper function to validate slots specified in GUC standby_slot_names.
+ *
+ * The rawname will be parsed, and the parsed result will be saved into
+ * *elemlist.
+ */
+static bool
+validate_standby_slots(char *rawname, List **elemlist)
+{
+ bool ok;
+
+ /* Verify syntax and parse string into a list of identifiers */
+ ok = SplitIdentifierString(rawname, ',', elemlist);
+
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
+ else if (!ReplicationSlotCtl)
+ {
+ /*
+ * We cannot validate the replication slot if the replication slots'
+ * data has not been initialized. This is ok as we will anyway
+ * validate the specified slot when waiting for them to catch up. See
+ * StandbySlotsHaveCaughtup() for details.
+ */
+ }
+ else
+ {
+ /* Check that the specified slots exist and are logical slots */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, *elemlist)
+ {
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ GUC_check_errdetail("replication slot \"%s\" does not exist",
+ name);
+ ok = false;
+ break;
+ }
+
+ if (!SlotIsPhysical(slot))
+ {
+ GUC_check_errdetail("\"%s\" is not a physical replication slot",
+ name);
+ ok = false;
+ break;
+ }
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+ }
+
+ return ok;
+}
+
+/*
+ * GUC check_hook for standby_slot_names
+ */
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ char *rawname;
+ char *ptr;
+ List *elemlist;
+ int size;
+ StandbySlotConfigData *config;
+
+ if ((*newval)[0] == '\0')
+ return true;
+
+ /*
+ * "*" is not accepted as in that case the primary will not be able to
+ * know for which all standbys to wait for. It is inappropriate to block
+ * logical replication for physical slots that either lack associated
+ * standbys or have standbys associated that are not enabled for
+ * replication slot synchronization.
+ */
+ if (strcmp(*newval, "*") == 0)
+ {
+ GUC_check_errdetail("\"*\" is not accepted for standby_slot_names");
+ return false;
+ }
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Now verify if the specified slots exist and have correct type */
+ if (!validate_standby_slots(rawname, &elemlist))
+ {
+ pfree(rawname);
+ list_free(elemlist);
+ return false;
+ }
+
+ /* Compute the size required for the StandbySlotConfigData struct */
+ size = offsetof(StandbySlotConfigData, slot_names);
+ foreach_ptr(char, slot_name, elemlist)
+ size += strlen(slot_name) + 1;
+
+ /* GUC extra value must be guc_malloc'd, not palloc'd */
+ config = (StandbySlotConfigData *) guc_malloc(LOG, size);
+
+ /* Transform the data into StandbySlotConfigData */
+ config->slot_num = list_length(elemlist);
+
+ ptr = config->slot_names;
+ foreach_ptr(char, slot_name, elemlist)
+ {
+ strcpy(ptr, slot_name);
+ ptr += strlen(slot_name) + 1;
+ }
+
+ *extra = (void *) config;
+
+ pfree(rawname);
+ list_free(elemlist);
+ return true;
+}
+
+/*
+ * GUC assign_hook for standby_slot_names
+ */
+void
+assign_standby_slot_names(const char *newval, void *extra)
+{
+ /*
+ * The standby slots may have changed, so we must recompute the oldest
+ * LSN.
+ */
+ ss_oldest_flush_lsn = InvalidXLogRecPtr;
+
+ standby_slot_config = (StandbySlotConfigData *) extra;
+}
+
+/*
+ * Return true if the currently acquired slot is specified in
+ * standby_slot_names GUC; otherwise, return false.
+ */
+bool
+AcquiredStandbySlot(void)
+{
+ const char *name;
+
+ /* Return false if there is no value in standby_slot_names */
+ if (standby_slot_config == NULL)
+ return false;
+
+ name = standby_slot_config->slot_names;
+ for (int i = 0; i < standby_slot_config->slot_num; i++)
+ {
+ if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0)
+ return true;
+
+ name += strlen(name) + 1;
+ }
+
+ return false;
+}
+
+/*
+ * Return true if the slots specified in standby_slot_names have caught up to
+ * the given WAL location, false otherwise.
+ *
+ * The elevel parameter specifies the error level used for logging messages
+ * related to slots that do not exist, are invalidated, or are inactive.
+ */
+bool
+StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel)
+{
+ const char *name;
+ int caught_up_slot_num = 0;
+ XLogRecPtr min_restart_lsn = InvalidXLogRecPtr;
+
+ /*
+ * Don't need to wait for the standbys to catch up if there is no value in
+ * standby_slot_names.
+ */
+ if (standby_slot_config == NULL)
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if we are on a standby
+ * server, since we do not support syncing slots to cascading standbys.
+ */
+ if (RecoveryInProgress())
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if they are already
+ * beyond the specified WAL location.
+ */
+ if (!XLogRecPtrIsInvalid(ss_oldest_flush_lsn) &&
+ ss_oldest_flush_lsn >= wait_for_lsn)
+ return true;
+
+ /*
+ * To prevent concurrent slot dropping and creation while filtering the
+ * slots, take the ReplicationSlotControlLock outside of the loop.
+ */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ name = standby_slot_config->slot_names;
+ for (int i = 0; i < standby_slot_config->slot_num; i++)
+ {
+ XLogRecPtr restart_lsn;
+ bool invalidated;
+ bool inactive;
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ /*
+ * If a slot name provided in standby_slot_names does not exist,
+ * report a message and exit the loop. A user can specify a slot
+ * name that does not exist just before the server startup. The
+ * GUC check_hook(validate_standby_slots) cannot validate such a
+ * slot during startup as the ReplicationSlotCtl shared memory is
+ * not initialized at that time. It is also possible for a user to
+ * drop the slot in standby_slot_names afterwards.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not exist",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider creating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (SlotIsLogical(slot))
+ {
+ /*
+ * If a logical slot name is provided in standby_slot_names,
+ * report a message and exit the loop. Similar to the non-existent
+ * case, a user can specify a logical slot name in
+ * standby_slot_names before the server startup, or drop an
+ * existing physical slot and recreate a logical slot with the
+ * same name.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot have logical replication slot \"%s\" in parameter %s",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting for correction on \"%s\".",
+ name),
+ errhint("Consider removing logical slot \"%s\" from parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ SpinLockAcquire(&slot->mutex);
+ restart_lsn = slot->data.restart_lsn;
+ invalidated = slot->data.invalidated != RS_INVAL_NONE;
+ inactive = slot->active_pid == 0;
+ SpinLockRelease(&slot->mutex);
+
+ if (invalidated)
+ {
+ /* Specified physical slot has been invalidated */
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("physical slot \"%s\" specified in parameter %s has been invalidated",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider dropping and recreating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (XLogRecPtrIsInvalid(restart_lsn) || restart_lsn < wait_for_lsn)
+ {
+ /* Log a message if no active_pid for this physical slot */
+ if (inactive)
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not have active_pid",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider starting standby associated with \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+
+ /* Continue if the current slot hasn't caught up. */
+ break;
+ }
+
+ Assert(restart_lsn >= wait_for_lsn);
+
+ if (XLogRecPtrIsInvalid(min_restart_lsn) ||
+ min_restart_lsn > restart_lsn)
+ min_restart_lsn = restart_lsn;
+
+ caught_up_slot_num++;
+
+ name += strlen(name) + 1;
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /*
+ * Return false if not all the standbys have caught up to the specified
+ * WAL location.
+ */
+ if (caught_up_slot_num != standby_slot_config->slot_num)
+ return false;
+
+ /* The ss_oldest_flush_lsn must not retreat. */
+ Assert(XLogRecPtrIsInvalid(ss_oldest_flush_lsn) ||
+ min_restart_lsn >= ss_oldest_flush_lsn);
+
+ ss_oldest_flush_lsn = min_restart_lsn;
+
+ return true;
+}
+
+/*
+ * Wait for physical standbys to confirm receiving the given lsn.
+ *
+ * Used by logical decoding SQL functions. It waits for physical standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+void
+WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
+{
+ /*
+ * Don't need to wait for the standby to catch up if the current acquired
+ * slot is not a logical failover slot, or there is no value in
+ * standby_slot_names.
+ */
+ if (!MyReplicationSlot->data.failover || !standby_slot_config)
+ return;
+
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+
+ for (;;)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /* Exit if done waiting for every slot. */
+ if (StandbySlotsHaveCaughtup(wait_for_lsn, WARNING))
+ break;
+
+ /*
+ * Wait for the slots in the standby_slot_names to catch up, but use a
+ * timeout (1s) so we can also check if the standby_slot_names has
+ * been changed.
+ */
+ ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000,
+ WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION);
+ }
+
+ ConditionVariableCancelSleep();
+}
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 768a304723..ad79e1fccd 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -464,6 +464,12 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* crash, but this makes the data consistent after a clean shutdown.
*/
ReplicationSlotMarkDirty();
+
+ /*
+ * Wake up logical walsenders holding logical failover slots after
+ * updating the restart_lsn of the physical slot.
+ */
+ PhysicalWakeupLogicalWalSnd();
}
return retlsn;
@@ -504,6 +510,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto)
.segment_close = wal_segment_close),
NULL, NULL, NULL);
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to moveto lsn.
+ */
+ WaitForStandbyConfirmation(moveto);
+
/*
* Start reading at the slot's restart_lsn, which we know to point to
* a valid record.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 0f1047179c..d8bdc65dc1 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1725,26 +1725,111 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
ProcessPendingWrites();
}
+/*
+ * Wake up the logical walsender processes with logical failover slots if the
+ * currently acquired physical slot is specified in standby_slot_names GUC.
+ */
+void
+PhysicalWakeupLogicalWalSnd(void)
+{
+ Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot));
+
+ /*
+ * If we are running in a standby, there is no need to wake up walsenders.
+ * This is because we do not support syncing slots to cascading standbys,
+ * so, there are no walsenders waiting for standbys to catch up.
+ */
+ if (RecoveryInProgress())
+ return;
+
+ if (AcquiredStandbySlot())
+ ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv);
+}
+
+/*
+ * Returns true if not all standbys have caught up to the flushed position
+ * (flushed_lsn) when the current acquired slot is a logical failover
+ * slot and we are streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForStandbys(XLogRecPtr flushed_lsn, uint32 *wait_event)
+{
+ int elevel = got_STOPPING ? ERROR : WARNING;
+ bool failover_slot;
+
+ failover_slot = (replication_active && MyReplicationSlot->data.failover);
+
+ /*
+ * Note that after receiving the shutdown signal, an ERROR is reported if
+ * any slots are dropped, invalidated, or inactive. This measure is taken
+ * to prevent the walsender from waiting indefinitely.
+ */
+ if (failover_slot && !StandbySlotsHaveCaughtup(flushed_lsn, elevel))
+ {
+ *wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
+ return true;
+ }
+
+ *wait_event = 0;
+ return false;
+}
+
+/*
+ * Returns true if we need to wait for WALs to be flushed to disk, or if not
+ * all standbys have caught up to the flushed position (flushed_lsn) when the
+ * current acquired slot is a logical failover slot and we are
+ * streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
+{
+ /* Check if we need to wait for WALs to be flushed to disk */
+ if (target_lsn > flushed_lsn)
+ {
+ *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
+ return true;
+ }
+
+ /* Check if the standby slots have caught up to the flushed position */
+ return NeedToWaitForStandbys(flushed_lsn, wait_event);
+}
+
/*
* Wait till WAL < loc is flushed to disk so it can be safely sent to client.
*
- * Returns end LSN of flushed WAL. Normally this will be >= loc, but
- * if we detect a shutdown request (either from postmaster or client)
- * we will return early, so caller must always check.
+ * If the walsender holds a logical slot that has enabled failover, we also
+ * wait for all the specified streaming replication standby servers to confirm
+ * receipt of WAL up to RecentFlushPtr. It is beneficial to wait here for the
+ * confirmation up to RecentFlushPtr rather than waiting before transmitting
+ * each change to logical subscribers, which is already covered by
+ * RecentFlushPtr.
+ *
+ * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we
+ * detect a shutdown request (either from postmaster or client) we will return
+ * early, so caller must always check.
*/
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
int wakeEvents;
+ uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
- * have enough WAL available. This is particularly interesting if we're
- * far behind.
+ * have enough WAL available and all the standby servers have confirmed
+ * receipt of WAL up to RecentFlushPtr. This is particularly interesting
+ * if we're far behind.
*/
- if (RecentFlushPtr != InvalidXLogRecPtr &&
- loc <= RecentFlushPtr)
+ if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
return RecentFlushPtr;
/* Get a more recent flush pointer. */
@@ -1753,8 +1838,14 @@ WalSndWaitForWal(XLogRecPtr loc)
else
RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * Within the loop, we wait for the necessary WALs to be flushed to disk
+ * first, followed by waiting for standbys to catch up if there are enough
+ * WALs (see NeedToWaitForWal()) or upon receiving the shutdown signal.
+ */
for (;;)
{
+ bool wait_for_standby_at_stop = false;
long sleeptime;
/* Clear any already-pending wakeups */
@@ -1781,21 +1872,35 @@ WalSndWaitForWal(XLogRecPtr loc)
if (got_STOPPING)
XLogBackgroundFlush();
- /* Update our idea of the currently flushed position. */
- if (!RecoveryInProgress())
- RecentFlushPtr = GetFlushRecPtr(NULL);
- else
- RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * To avoid the scenario where standbys need to catch up to a newer
+ * WAL location in each iteration, we update our idea of the currently
+ * flushed position only if we are not waiting for standbys to catch
+ * up.
+ */
+ if (wait_event != WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ {
+ if (!RecoveryInProgress())
+ RecentFlushPtr = GetFlushRecPtr(NULL);
+ else
+ RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ }
/*
- * If postmaster asked us to stop, don't wait anymore.
+ * If postmaster asked us to stop and the standby slots have caught up
+ * to the flushed position, don't wait anymore.
*
* It's important to do this check after the recomputation of
* RecentFlushPtr, so we can send all remaining data before shutting
* down.
*/
if (got_STOPPING)
- break;
+ {
+ if (NeedToWaitForStandbys(RecentFlushPtr, &wait_event))
+ wait_for_standby_at_stop = true;
+ else
+ break;
+ }
/*
* We only send regular messages to the client for full decoded
@@ -1810,11 +1915,18 @@ WalSndWaitForWal(XLogRecPtr loc)
!waiting_for_ping_response)
WalSndKeepalive(false, InvalidXLogRecPtr);
- /* check whether we're done */
- if (loc <= RecentFlushPtr)
+ /*
+ * Exit the loop if already caught up and doesn't need to wait for
+ * standby slots.
+ */
+ if (!wait_for_standby_at_stop &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
break;
- /* Waiting for new WAL. Since we need to wait, we're now caught up. */
+ /*
+ * Waiting for new WAL or waiting for standbys to catch up. Since we
+ * need to wait, we're now caught up.
+ */
WalSndCaughtUp = true;
/*
@@ -1852,7 +1964,9 @@ WalSndWaitForWal(XLogRecPtr loc)
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL);
+ Assert(wait_event != 0);
+
+ WalSndWait(wakeEvents, sleeptime, wait_event);
}
/* reactivate latch so WalSndLoop knows to continue */
@@ -2262,6 +2376,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn)
{
ReplicationSlotMarkDirty();
ReplicationSlotsComputeRequiredLSN();
+ PhysicalWakeupLogicalWalSnd();
}
/*
@@ -3535,6 +3650,7 @@ WalSndShmemInit(void)
ConditionVariableInit(&WalSndCtl->wal_flush_cv);
ConditionVariableInit(&WalSndCtl->wal_replay_cv);
+ ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv);
}
}
@@ -3604,8 +3720,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event)
*
* And, we use separate shared memory CVs for physical and logical
* walsenders for selective wake ups, see WalSndWakeup() for more details.
+ *
+ * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV
+ * until awakened by physical walsenders after the walreceiver confirms
+ * the receipt of the LSN.
*/
- if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
+ if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+ else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv);
else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index ec2f31f82a..c08e00d1d6 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP
LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server."
LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server."
SSL_OPEN_SERVER "Waiting for SSL while attempting connection."
+WAIT_FOR_STANDBY_CONFIRMATION "Waiting for WAL to be received and flushed by the physical standby."
WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process."
WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 45013582a7..d77214795d 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4670,6 +4670,20 @@ struct config_string ConfigureNamesString[] =
check_debug_io_direct, assign_debug_io_direct, NULL
},
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("Lists streaming replication standby server slot "
+ "names that logical WAL sender processes will wait for."),
+ gettext_noop("Logical WAL sender processes will send decoded "
+ "changes to plugins only after the specified "
+ "replication slots confirm receiving WAL."),
+ GUC_LIST_INPUT
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, assign_standby_slot_names, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index edcc0282b2..2244ee52f7 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -343,6 +343,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsender processes will wait for
# - Standby Servers -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index acbf567150..73f7154173 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -226,6 +226,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
+extern PGDLLIMPORT char *standby_slot_names;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -274,4 +275,8 @@ extern void CheckSlotPermissions(void);
extern ReplicationSlotInvalidationCause
GetSlotInvalidationCause(const char *conflict_reason);
+extern bool AcquiredStandbySlot(void);
+extern bool StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel);
+extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn);
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index 0c3996e926..f2d8297f01 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -39,6 +39,7 @@ extern void InitWalSender(void);
extern bool exec_replication_command(const char *cmd_string);
extern void WalSndErrorCleanup(void);
extern void WalSndResourceCleanup(bool isCommit);
+extern void PhysicalWakeupLogicalWalSnd(void);
extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
extern void WalSndSignals(void);
extern Size WalSndShmemSize(void);
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 3113e9ea47..109924ffcd 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -113,6 +113,13 @@ typedef struct
ConditionVariable wal_flush_cv;
ConditionVariable wal_replay_cv;
+ /*
+ * Used by physical walsenders holding slots specified in
+ * standby_slot_names to wake up logical walsenders holding logical
+ * failover slots when a walreceiver confirms the receipt of LSN.
+ */
+ ConditionVariable wal_confirm_rcv_cv;
+
WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER];
} WalSndCtlData;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index c8a7aa9a11..d64dc5fcdb 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra,
extern void assign_wal_consistency_checking(const char *newval, void *extra);
extern bool check_wal_segment_size(int *newval, void **extra, GucSource source);
extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
+extern bool check_standby_slot_names(char **newval, void **extra,
+ GucSource source);
+extern void assign_standby_slot_names(const char *newval, void *extra);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl
index 5c7b4ca5e3..b95d95c06f 100644
--- a/src/test/recovery/t/006_logical_decoding.pl
+++ b/src/test/recovery/t/006_logical_decoding.pl
@@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'},
undef, 'logical slot was actually dropped with DB');
# Test logical slot advancing and its durability.
+# Passing failover=true (last arg) should not have any impact on advancing.
my $logical_slot = 'logical_slot';
$node_primary->safe_psql('postgres',
- "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);"
+ "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);"
);
$node_primary->psql(
'postgres', "
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 021c58f621..99025b352a 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -504,18 +504,258 @@ ok( $standby1->poll_query_until(
"SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot' AND synced AND NOT temporary;"),
'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby');
+##################################################
+# Test that logical failover replication slots wait for the specified
+# physical replication slots to receive the changes first. It uses the
+# following set up:
+#
+# (physical standbys)
+# | ----> standby1 (primary_slot_name = sb1_slot)
+# | ----> standby2 (primary_slot_name = sb2_slot)
+# primary ----- |
+# (logical replication)
+# | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
+# | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
+#
+# standby_slot_names = 'sb1_slot'
+#
+# The setup is configured in such a way that the logical slot of subscriber1 is
+# enabled for failover, and thus the subscriber1 will wait for the physical
+# slot of standby1(sb1_slot) to catch up before receiving the decoded changes.
+##################################################
+
+$backup_name = 'backup3';
+
+$primary->psql('postgres',
+ q{SELECT pg_create_physical_replication_slot('sb2_slot');});
+
+$primary->backup($backup_name);
+
+# Create another standby
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup(
+ $primary, $backup_name,
+ has_streaming => 1,
+ has_restoring => 1);
+$standby2->append_conf(
+ 'postgresql.conf', qq(
+primary_slot_name = 'sb2_slot'
+));
+$standby2->start;
+$primary->wait_for_replay_catchup($standby2);
+
+# Configure primary to disallow any logical slots that have enabled failover
+# from getting ahead of the specified physical replication slot (sb1_slot).
+$primary->append_conf(
+ 'postgresql.conf', qq(
+standby_slot_names = 'sb1_slot'
+));
+$primary->reload;
+
+# Create another subscriber node without enabling failover, wait for sync to
+# complete
+my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$subscriber2->init;
+$subscriber2->start;
+$subscriber2->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE tab_int (a int PRIMARY KEY);
+ CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot);
+]);
+
+$subscriber2->wait_for_subscription_sync;
+
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+my $offset = -s $primary->logfile;
+
+# Stop the standby associated with the specified physical replication slot
+# (sb1_slot) so that the logical replication slot (lsub1_slot) won't receive
+# changes until the standby comes up.
+$standby1->stop;
+
+# Create some data on the primary
+my $primary_row_count = 20;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(11, $primary_row_count);");
+
+# Wait until the standby2 that's still running gets the data from the primary
+$primary->wait_for_replay_catchup($standby2);
+$result = $standby2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby2 gets data from primary");
+
+# Wait for regress_mysub2 to get the data from the primary. This subscription
+# was not enabled for failover so it gets the data without waiting for any
+# standbys.
+$primary->wait_for_catchup('regress_mysub2');
+$result = $subscriber2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "subscriber2 gets data from primary");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# The regress_mysub1 was enabled for failover so it doesn't get the data from
+# primary and keeps waiting for the standby specified in standby_slot_names
+# (sb1_slot aka standby1).
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) <> $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data from primary until standby1 acknowledges changes"
+);
+
+# Start the standby specified in standby_slot_names (sb1_slot aka standby1) and
+# wait for it to catch up with the primary.
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+$result = $standby1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby1 gets data from primary");
+
+# Now that the standby specified in standby_slot_names is up and running, the
+# primary can send the decoded changes to the subscription enabled for failover
+# (i.e. regress_mysub1). While the standby was down, regress_mysub1 didn't
+# receive any data from the primary. i.e. the primary didn't allow it to go
+# ahead of standby.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 acknowledges changes");
+
+##################################################
+# Verify that when using pg_logical_slot_get_changes to consume changes from a
+# logical slot with failover enabled, it will also wait for the slots specified
+# in standby_slot_names to catch up.
+##################################################
+
+# Stop the standby associated with the specified physical replication slot so
+# that the logical replication slot won't receive changes until the standby
+# slot's restart_lsn is advanced or the slot is removed from the
+# standby_slot_names list.
+$primary->safe_psql('postgres', "TRUNCATE tab_int;");
+$primary->wait_for_catchup('regress_mysub1');
+$standby1->stop;
+
+# Disable the regress_mysub1 to prevent the logical walsender from generating
+# more warnings.
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 DISABLE");
+
+# Wait for the replication slot to become inactive on the publisher
+$primary->poll_query_until(
+ 'postgres',
+ "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'",
+ 1);
+
+# Create a logical 'test_decoding' replication slot with failover enabled
+$primary->safe_psql('postgres',
+ "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);"
+);
+
+my $back_q = $primary->background_psql(
+ 'postgres',
+ on_error_stop => 0,
+ timeout => $PostgreSQL::Test::Utils::timeout_default);
+
+# pg_logical_slot_get_changes will be blocked until the standby catches up,
+# hence it needs to be executed in a background session.
+$offset = -s $primary->logfile;
+$back_q->query_until(
+ qr/logical_slot_get_changes/, q(
+ \echo logical_slot_get_changes
+ SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);
+));
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the function
+# pg_logical_slot_get_changes should now return, and the session can be
+# stopped.
+$back_q->quit;
+
+$primary->safe_psql('postgres',
+ "SELECT pg_drop_replication_slot('test_slot');"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
+# Enable the regress_mysub1 for further tests
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+##################################################
+# Test that logical replication will wait for the user-created inactive
+# physical slot to catch up until we remove the slot from standby_slot_names.
+##################################################
+
+$offset = -s $primary->logfile;
+
+# Create some data on the primary
+$primary_row_count = 10;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# The regress_mysub1 doesn't get the data from primary because the specified
+# standby slot (sb1_slot) in standby_slot_names is inactive.
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data as the sb1_slot doesn't catch up");
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the primary server should now
+# send the decoded changes to the subscription.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
##################################################
# Promote the standby1 to primary. Confirm that:
# a) the slot 'lsub1_slot' is retained on the new primary
# b) logical replication for regress_mysub1 is resumed successfully after failover
##################################################
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+
$standby1->promote;
# Update subscription with the new primary's connection info
my $standby1_conninfo = $standby1->connstr . ' dbname=postgres';
$subscriber1->safe_psql('postgres',
- "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo';
- ALTER SUBSCRIPTION regress_mysub1 ENABLE; ");
+ "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo';");
# Confirm the synced slot 'lsub1_slot' is retained on the new primary
is($standby1->safe_psql('postgres',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 95ae7845d8..c0b9875838 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2650,6 +2650,7 @@ SplitPoint
SplitTextOutputData
SplitVar
StackElem
+StandbySlotConfigData
StartDataPtrType
StartLOPtrType
StartLOsPtrType
--
2.30.0.windows.2
[application/octet-stream] v107_2-0002-Document-the-steps-to-check-if-the-standby-is-r.patch (7.0K, ../../OS0PR01MB57161D93206707DAF900C34F94212@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v107_2-0002-Document-the-steps-to-check-if-the-standby-is-r.patch)
download | inline diff:
From 94ed723efb005142940925af70fa577c3877c732 Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Fri, 19 Jan 2024 11:04:16 +0530
Subject: [PATCH v107 2/2] Document the steps to check if the standby is ready
for failover
---
doc/src/sgml/high-availability.sgml | 9 ++
doc/src/sgml/logical-replication.sgml | 136 ++++++++++++++++++++++++++
2 files changed, 145 insertions(+)
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 236c0af65f..36215aa68c 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -1487,6 +1487,15 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)'
Written administration procedures are advised.
</para>
+ <para>
+ If you have opted for synchronization of logical slots (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>),
+ then before switching to the standby server, it is recommended to check
+ if the logical slots synchronized on the standby server are ready
+ for failover. This can be done by following the steps described in
+ <xref linkend="logical-replication-failover"/>.
+ </para>
+
<para>
To trigger failover of a log-shipping standby server, run
<command>pg_ctl promote</command> or call <function>pg_promote()</function>.
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ec2130669e..be59d306a1 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -687,6 +687,142 @@ ALTER SUBSCRIPTION
</sect1>
+ <sect1 id="logical-replication-failover">
+ <title>Logical Replication Failover</title>
+
+ <para>
+ When the publisher server is the primary server of a streaming replication,
+ the logical slots on that primary server can be synchronized to the standby
+ server by specifying <literal>failover = true</literal> when creating
+ subscriptions for those publications. Enabling failover ensures a seamless
+ transition of those subscriptions after the standby is promoted. They can
+ continue subscribing to publications now on the new primary server without
+ any data loss.
+ </para>
+
+ <para>
+ Because the slot synchronization logic copies asynchronously, it is
+ necessary to confirm that replication slots have been synced to the standby
+ server before the failover happens. Furthermore, to ensure a successful
+ failover, the standby server must not be lagging behind the subscriber. It
+ is highly recommended to use
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ to prevent the subscriber from consuming changes faster than the hot standby.
+ To confirm that the standby server is indeed ready for failover, follow
+ these 2 steps:
+ </para>
+
+ <procedure>
+ <step performance="required">
+ <para>
+ Confirm that all the necessary logical replication slots have been synced to
+ the standby server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node, use the following SQL to identify
+ which slots should be synced to the standby that we plan to promote.
+<programlisting>
+test_sub=# SELECT
+ array_agg(slotname) AS slots
+ FROM
+ ((
+ SELECT r.srsubid AS subid, CONCAT('pg_', srsubid, '_sync_', srrelid, '_', ctl.system_identifier) AS slotname
+ FROM pg_control_system() ctl, pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate = 'f' AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT s.oid AS subid, s.subslotname as slotname
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ slots
+-------
+ {sub1,sub2,sub3}
+(1 row)
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, check that the logical replication slots identified above exist on
+ the standby server and are ready for failover.
+<programlisting>
+test_standby=# SELECT slot_name, (synced AND NOT temporary AND conflict_reason IS NULL) AS failover_ready
+ FROM pg_replication_slots
+ WHERE slot_name IN ('sub1','sub2','sub3');
+ slot_name | failover_ready
+-------------+----------------
+ sub1 | t
+ sub2 | t
+ sub3 | t
+(3 rows)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+
+ <step performance="required">
+ <para>
+ Confirm that the standby server is not lagging behind the subscribers.
+ This step can be skipped if
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ has been correctly configured. If standby_slot_names is not configured
+ correctly, it is highly recommended to run this step after the primary
+ server is down, otherwise the results of the query may vary at different
+ points of time due to the ongoing replication on the logical subscribers
+ from the primary server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node check the last replayed WAL.
+ This step needs to be run on the database(s) that includes the failover
+ enabled subscription(s), to find the last replayed WAL on each database.
+<programlisting>
+test_sub=# SELECT
+ MAX(remote_lsn) AS remote_lsn_on_subscriber
+ FROM
+ ((
+ SELECT (CASE WHEN r.srsubstate = 'f' THEN pg_replication_origin_progress(CONCAT('pg_', r.srsubid, '_', r.srrelid), false)
+ WHEN r.srsubstate IN ('s', 'r') THEN r.srsublsn END) AS remote_lsn
+ FROM pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate IN ('f', 's', 'r') AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT pg_replication_origin_progress(CONCAT('pg_', s.oid), false) AS remote_lsn
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ remote_lsn_on_subscriber
+--------------------------
+ 0/3000388
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, on the standby server check that the last-received WAL location
+ is ahead of the replayed WAL location(s) on the subscriber identified
+ above. If the above SQL result was NULL, it means the subscriber has not
+ yet replayed any WAL, so the standby server must be ahead of the
+ subscriber, and this step can be skipped.
+<programlisting>
+test_standby=# SELECT pg_last_wal_receive_lsn() >= '0/3000388'::pg_lsn AS failover_ready;
+ failover_ready
+----------------
+ t
+(1 row)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+ </procedure>
+
+ <para>
+ If the result (<literal>failover_ready</literal>) of both above steps is
+ true, existing subscriptions will be able to continue without data loss.
+ </para>
+
+ </sect1>
+
<sect1 id="logical-replication-row-filter">
<title>Row Filters</title>
--
2.30.0.windows.2
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-07 02:04 Peter Smith <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
1 sibling, 2 replies; 68+ messages in thread
From: Peter Smith @ 2024-03-07 02:04 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Amit Kapila <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
Here are some review comments for v107-0001
======
src/backend/replication/slot.c
1.
+/*
+ * Struct for the configuration of standby_slot_names.
+ *
+ * Note: this must be a flat representation that can be held in a single chunk
+ * of guc_malloc'd memory, so that it can be stored as the "extra" data for the
+ * standby_slot_names GUC.
+ */
+typedef struct
+{
+ int slot_num;
+
+ /* slot_names contains nmembers consecutive nul-terminated C strings */
+ char slot_names[FLEXIBLE_ARRAY_MEMBER];
+} StandbySlotConfigData;
+
1a.
To avoid any ambiguity this 1st field is somehow a slot ID number, I
felt a better name would be 'nslotnames' or even just 'n' or 'count',
~
1b.
(fix typo)
SUGGESTION for the 2nd field comment
slot_names is a chunk of 'n' X consecutive null-terminated C strings
~
1c.
A more explanatory name for this typedef maybe is 'StandbySlotNamesConfigData' ?
~~~
2.
+/* This is parsed and cached configuration for standby_slot_names */
+static StandbySlotConfigData *standby_slot_config;
2a.
/This is parsed and cached configuration for .../This is the parsed
and cached configuration for .../
~
2b.
Similar to above -- since this only has name information maybe it is
more correct to call it 'standby_slot_names_config'?
~~~
3.
+/*
+ * A helper function to validate slots specified in GUC standby_slot_names.
+ *
+ * The rawname will be parsed, and the parsed result will be saved into
+ * *elemlist.
+ */
+static bool
+validate_standby_slots(char *rawname, List **elemlist)
/and the parsed result/and the result/
~~~
4. check_standby_slot_names
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
/copy of string/copy of the GUC string/
~~~
5.
+assign_standby_slot_names(const char *newval, void *extra)
+{
+ /*
+ * The standby slots may have changed, so we must recompute the oldest
+ * LSN.
+ */
+ ss_oldest_flush_lsn = InvalidXLogRecPtr;
+
+ standby_slot_config = (StandbySlotConfigData *) extra;
+}
To avoid leaking don't we need to somewhere take care to free any
memory used by a previous value (if any) of this
'standby_slot_config'?
~~~
6. AcquiredStandbySlot
+/*
+ * Return true if the currently acquired slot is specified in
+ * standby_slot_names GUC; otherwise, return false.
+ */
+bool
+AcquiredStandbySlot(void)
+{
+ const char *name;
+
+ /* Return false if there is no value in standby_slot_names */
+ if (standby_slot_config == NULL)
+ return false;
+
+ name = standby_slot_config->slot_names;
+ for (int i = 0; i < standby_slot_config->slot_num; i++)
+ {
+ if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0)
+ return true;
+
+ name += strlen(name) + 1;
+ }
+
+ return false;
+}
6a.
Just checking "(standby_slot_config == NULL)" doesn't seem enough to
me, because IIUIC it is possible when 'standby_slot_names' has no
value then maybe standby_slot_config is not NULL but
standby_slot_config->slot_num is 0.
~
6b.
IMO this function would be tidier written such that the
MyReplicationSlot->data.name is passed as a parameter. Then you can
name the function more naturally like:
IsSlotInStandbySlotNames(const char *slot_name)
~
6c.
IMO the body of the function will be tidier if written so there are
only 2 returns instead of 3 like
SUGGESTION:
if (...)
{
for (...)
{
...
return true;
}
}
return false;
~~~
7.
+ /*
+ * Don't need to wait for the standbys to catch up if there is no value in
+ * standby_slot_names.
+ */
+ if (standby_slot_config == NULL)
+ return true;
(similar to a previous review comment)
This check doesn't seem enough because IIUIC it is possible when
'standby_slot_names' has no value then maybe standby_slot_config is
not NULL but standby_slot_config->slot_num is 0.
~~~
8. WaitForStandbyConfirmation
+ /*
+ * Don't need to wait for the standby to catch up if the current acquired
+ * slot is not a logical failover slot, or there is no value in
+ * standby_slot_names.
+ */
+ if (!MyReplicationSlot->data.failover || !standby_slot_config)
+ return;
(similar to a previous review comment)
IIUIC it is possible that when 'standby_slot_names' has no value, then
standby_slot_config is not NULL but standby_slot_config->slot_num is
0. So shouldn't that be checked too?
Perhaps it is convenient to encapsulate this check using some macro:
#define StandbySlotNamesHasNoValue() (standby_slot_config = NULL ||
standby_slot_config->slot_num == 0)
----------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-07 03:07 shveta malik <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
1 sibling, 1 reply; 68+ messages in thread
From: shveta malik @ 2024-03-07 03:07 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>; shveta malik <[email protected]>
On Wed, Mar 6, 2024 at 6:54 PM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> On Wednesday, March 6, 2024 9:13 PM Zhijie Hou (Fujitsu) <[email protected]> wrote:
> >
> > On Wednesday, March 6, 2024 11:04 AM Zhijie Hou (Fujitsu)
> > <[email protected]> wrote:
> > >
> > > On Wednesday, March 6, 2024 9:30 AM Masahiko Sawada
> > > <[email protected]> wrote:
> > >
> > > Hi,
> > >
> > > > On Fri, Mar 1, 2024 at 4:21 PM Zhijie Hou (Fujitsu)
> > > > <[email protected]>
> > > > wrote:
> > > > >
> > > > > On Friday, March 1, 2024 2:11 PM Masahiko Sawada
> > > > <[email protected]> wrote:
> > > > > >
> > > > > >
> > > > > > ---
> > > > > > +void
> > > > > > +assign_standby_slot_names(const char *newval, void *extra) {
> > > > > > + List *standby_slots;
> > > > > > + MemoryContext oldcxt;
> > > > > > + char *standby_slot_names_cpy = extra;
> > > > > > +
> > > > > >
> > > > > > Given that the newval and extra have the same data
> > > > > > (standby_slot_names value), why do we not use newval instead? I
> > > > > > think that if we use newval, we don't need to guc_strdup() in
> > > > > > check_standby_slot_names(), we might need to do list_copy_deep()
> > > > > > instead, though. It's not clear to me as there is no comment.
> > > > >
> > > > > I think SplitIdentifierString will modify the passed in string, so
> > > > > we'd better not pass the newval to it, otherwise the stored guc
> > > > > string(standby_slot_names) will be changed. I can see we are doing
> > > > > similar thing in other GUC check/assign function as well.
> > > > > (check_wal_consistency_checking/ assign_wal_consistency_checking,
> > > > > check_createrole_self_grant/ assign_createrole_self_grant ...).
> > > >
> > > > Why does it have to be a List in the first place?
> > >
> > > I thought the List type is convenient to use here, as we have existing
> > > list build function(SplitIdentifierString), and have convenient list
> > > macro to loop the
> > > list(foreach_ptr) which can save some codes.
> > >
> > > > In earlier version patches, we
> > > > used to copy the list and delete the element until it became empty,
> > > > while waiting for physical wal senders. But we now just refer to
> > > > each slot name in the list. The current code assumes that
> > > > stnadby_slot_names_cpy is allocated in GUCMemoryContext but once it
> > > > changes, it will silently get broken. I think we can check and
> > > > assign standby_slot_names in a similar way to
> > > > check/assign_temp_tablespaces and
> > check/assign_synchronous_standby_names.
> > >
> > > Yes, we could do follow it by allocating an array and copy each slot
> > > name into it, but it also requires some codes to build and scan the
> > > array. So, is it possible to expose the GucMemorycontext or have an API like
> > guc_copy_list instead ?
> > > If we don't want to touch the guc api, I am ok with using an array as well.
> >
> > I rethink about this and realize that it's not good to do the memory allocation in
> > assign hook function. As the "src/backend/utils/misc/README" said, we'd
> > better do that in check hook function and pass it via extra to assign hook
> > function. And thus array is a good choice in this case rather than a List which
> > cannot be passed to *extra.
> >
> > Here is the V107 patch set which parse and cache the standby slot names in an
> > array instead of a List.
>
> The patch needs to be rebased due to recent commit.
>
> Attach the V107_2 path set. There are no code changes in this version.
The patch needed to be rebased due to a recent commit. Attached
v107_3, there are no code changes in this version.
thanks
Shveta
Attachments:
[application/octet-stream] v107_3-0002-Document-the-steps-to-check-if-the-standby-is.patch (7.0K, ../../CAJpy0uBONK2E7360GtXLjpZwrOZeYE1vX=5Wv+gF8ZVB_OVLBQ@mail.gmail.com/2-v107_3-0002-Document-the-steps-to-check-if-the-standby-is.patch)
download | inline diff:
From 85d9d3ad26058da9d277b136be76359033f18a43 Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Fri, 19 Jan 2024 11:04:16 +0530
Subject: [PATCH v107_3 2/2] Document the steps to check if the standby is
ready for failover
---
doc/src/sgml/high-availability.sgml | 9 ++
doc/src/sgml/logical-replication.sgml | 136 ++++++++++++++++++++++++++
2 files changed, 145 insertions(+)
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 236c0af65f..36215aa68c 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -1487,6 +1487,15 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)'
Written administration procedures are advised.
</para>
+ <para>
+ If you have opted for synchronization of logical slots (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>),
+ then before switching to the standby server, it is recommended to check
+ if the logical slots synchronized on the standby server are ready
+ for failover. This can be done by following the steps described in
+ <xref linkend="logical-replication-failover"/>.
+ </para>
+
<para>
To trigger failover of a log-shipping standby server, run
<command>pg_ctl promote</command> or call <function>pg_promote()</function>.
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ec2130669e..be59d306a1 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -687,6 +687,142 @@ ALTER SUBSCRIPTION
</sect1>
+ <sect1 id="logical-replication-failover">
+ <title>Logical Replication Failover</title>
+
+ <para>
+ When the publisher server is the primary server of a streaming replication,
+ the logical slots on that primary server can be synchronized to the standby
+ server by specifying <literal>failover = true</literal> when creating
+ subscriptions for those publications. Enabling failover ensures a seamless
+ transition of those subscriptions after the standby is promoted. They can
+ continue subscribing to publications now on the new primary server without
+ any data loss.
+ </para>
+
+ <para>
+ Because the slot synchronization logic copies asynchronously, it is
+ necessary to confirm that replication slots have been synced to the standby
+ server before the failover happens. Furthermore, to ensure a successful
+ failover, the standby server must not be lagging behind the subscriber. It
+ is highly recommended to use
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ to prevent the subscriber from consuming changes faster than the hot standby.
+ To confirm that the standby server is indeed ready for failover, follow
+ these 2 steps:
+ </para>
+
+ <procedure>
+ <step performance="required">
+ <para>
+ Confirm that all the necessary logical replication slots have been synced to
+ the standby server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node, use the following SQL to identify
+ which slots should be synced to the standby that we plan to promote.
+<programlisting>
+test_sub=# SELECT
+ array_agg(slotname) AS slots
+ FROM
+ ((
+ SELECT r.srsubid AS subid, CONCAT('pg_', srsubid, '_sync_', srrelid, '_', ctl.system_identifier) AS slotname
+ FROM pg_control_system() ctl, pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate = 'f' AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT s.oid AS subid, s.subslotname as slotname
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ slots
+-------
+ {sub1,sub2,sub3}
+(1 row)
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, check that the logical replication slots identified above exist on
+ the standby server and are ready for failover.
+<programlisting>
+test_standby=# SELECT slot_name, (synced AND NOT temporary AND conflict_reason IS NULL) AS failover_ready
+ FROM pg_replication_slots
+ WHERE slot_name IN ('sub1','sub2','sub3');
+ slot_name | failover_ready
+-------------+----------------
+ sub1 | t
+ sub2 | t
+ sub3 | t
+(3 rows)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+
+ <step performance="required">
+ <para>
+ Confirm that the standby server is not lagging behind the subscribers.
+ This step can be skipped if
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ has been correctly configured. If standby_slot_names is not configured
+ correctly, it is highly recommended to run this step after the primary
+ server is down, otherwise the results of the query may vary at different
+ points of time due to the ongoing replication on the logical subscribers
+ from the primary server.
+ </para>
+ <substeps>
+ <step performance="required">
+ <para>
+ Firstly, on the subscriber node check the last replayed WAL.
+ This step needs to be run on the database(s) that includes the failover
+ enabled subscription(s), to find the last replayed WAL on each database.
+<programlisting>
+test_sub=# SELECT
+ MAX(remote_lsn) AS remote_lsn_on_subscriber
+ FROM
+ ((
+ SELECT (CASE WHEN r.srsubstate = 'f' THEN pg_replication_origin_progress(CONCAT('pg_', r.srsubid, '_', r.srrelid), false)
+ WHEN r.srsubstate IN ('s', 'r') THEN r.srsublsn END) AS remote_lsn
+ FROM pg_subscription_rel r, pg_subscription s
+ WHERE r.srsubstate IN ('f', 's', 'r') AND s.oid = r.srsubid AND s.subfailover
+ ) UNION (
+ SELECT pg_replication_origin_progress(CONCAT('pg_', s.oid), false) AS remote_lsn
+ FROM pg_subscription s
+ WHERE s.subfailover
+ ));
+ remote_lsn_on_subscriber
+--------------------------
+ 0/3000388
+</programlisting></para>
+ </step>
+ <step performance="required">
+ <para>
+ Next, on the standby server check that the last-received WAL location
+ is ahead of the replayed WAL location(s) on the subscriber identified
+ above. If the above SQL result was NULL, it means the subscriber has not
+ yet replayed any WAL, so the standby server must be ahead of the
+ subscriber, and this step can be skipped.
+<programlisting>
+test_standby=# SELECT pg_last_wal_receive_lsn() >= '0/3000388'::pg_lsn AS failover_ready;
+ failover_ready
+----------------
+ t
+(1 row)
+</programlisting></para>
+ </step>
+ </substeps>
+ </step>
+ </procedure>
+
+ <para>
+ If the result (<literal>failover_ready</literal>) of both above steps is
+ true, existing subscriptions will be able to continue without data loss.
+ </para>
+
+ </sect1>
+
<sect1 id="logical-replication-row-filter">
<title>Row Filters</title>
--
2.34.1
[application/octet-stream] v107_3-0001-Allow-logical-walsenders-to-wait-for-the-phys.patch (49.9K, ../../CAJpy0uBONK2E7360GtXLjpZwrOZeYE1vX=5Wv+gF8ZVB_OVLBQ@mail.gmail.com/3-v107_3-0001-Allow-logical-walsenders-to-wait-for-the-phys.patch)
download | inline diff:
From 5288e4438b7de7294b6a8b02f58cc8a2a56862ed Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Thu, 7 Mar 2024 08:28:52 +0530
Subject: [PATCH v107_3 1/2] Allow logical walsenders to wait for the physical
standby
This patch introduces a mechanism to ensure that physical standby servers,
which are potential failover candidates, have received and flushed changes
before making them visible to subscribers. By doing so, it guarantees that
the promoted standby server is not lagging behind the subscribers when a
failover is necessary.
A new parameter named 'standby_slot_names' is introduced. The logical
walsender now guarantees that all local changes are sent and flushed to
the standby servers corresponding to the replication slots specified in
'standby_slot_names' before sending those changes to the subscriber.
Additionally, The SQL functions pg_logical_slot_get_changes,
pg_logical_slot_peek_changes and pg_replication_slot_advance are modified.
Now, when used with logical failover slots, these functions will
block until all physical slots specified in 'standby_slot_names' have
confirmed WAL receipt.
---
doc/src/sgml/config.sgml | 45 ++
doc/src/sgml/func.sgml | 16 +-
doc/src/sgml/logicaldecoding.sgml | 12 +
.../replication/logical/logicalfuncs.c | 12 +
src/backend/replication/logical/slotsync.c | 11 +
src/backend/replication/slot.c | 407 +++++++++++++++++-
src/backend/replication/slotfuncs.c | 12 +
src/backend/replication/walsender.c | 160 ++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 14 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/replication/slot.h | 5 +
src/include/replication/walsender.h | 1 +
src/include/replication/walsender_private.h | 7 +
src/include/utils/guc_hooks.h | 3 +
src/test/recovery/t/006_logical_decoding.pl | 3 +-
.../t/040_standby_failover_slots_sync.pl | 244 ++++++++++-
src/tools/pgindent/typedefs.list | 1 +
18 files changed, 929 insertions(+), 27 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index b38cbd714a..792ab22290 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4559,6 +4559,51 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Lists the streaming replication standby server slot names that logical
+ WAL sender processes will wait for. Logical WAL sender processes will
+ send decoded changes to plugins only after the specified replication
+ slots confirm receiving WAL. This guarantees that logical replication
+ slots with failover enabled do not consume changes until those changes
+ are received and flushed to corresponding physical standbys. If a
+ logical replication connection is meant to switch to a physical standby
+ after the standby is promoted, the physical replication slot for the
+ standby should be listed here. Note that logical replication will not
+ proceed if the slots specified in the
+ <varname>standby_slot_names</varname> do not exist or are invalidated.
+ Additionally, the replication management functions
+ <link linkend="pg-replication-slot-advance">
+ <function>pg_replication_slot_advance</function></link>,
+ <link linkend="pg-logical-slot-get-changes">
+ <function>pg_logical_slot_get_changes</function></link>, and
+ <link linkend="pg-logical-slot-peek-changes">
+ <function>pg_logical_slot_peek_changes</function></link>,
+ when used with logical failover slots, will block until all
+ physical slots specified in <varname>standby_slot_names</varname> have
+ confirmed WAL receipt.
+ </para>
+ <para>
+ The standbys corresponding to the physical replication slots in
+ <varname>standby_slot_names</varname> must configure
+ <literal>sync_replication_slots = true</literal> so they can receive
+ logical failover slot changes from the primary.
+ </para>
+ <note>
+ <para>
+ The slots must be named explicitly. For example, specifying wildcard
+ values like <literal>*</literal> is not permitted.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index e5fa82c161..0bb7aeb40e 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -28150,7 +28150,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-get-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_get_changes</primary>
</indexterm>
@@ -28173,11 +28173,15 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
the specified value. Note, however, that the actual number of
rows returned may be larger, since this limit is only checked after
adding the rows produced when decoding each new transaction commit.
+ If the specified slot is a logical failover slot then the function will
+ not return until all physical slots specified in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ have confirmed WAL receipt.
</para></entry>
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-peek-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_peek_changes</primary>
</indexterm>
@@ -28232,7 +28236,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-replication-slot-advance" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_replication_slot_advance</primary>
</indexterm>
@@ -28248,7 +28252,11 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
the name of the slot and the actual position that it was advanced to.
The updated slot position information is written out at the next
checkpoint if any advancing is done. So in the event of a crash, the
- slot may return to an earlier position.
+ slot may return to an earlier position. If the specified slot is a
+ logical failover slot then the function will not return until all
+ physical slots specified in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ have confirmed WAL receipt.
</para></entry>
</row>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8f13780e74..5a15bbc580 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -384,6 +384,18 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU
must be enabled on the standby. It is also necessary to specify a valid
<literal>dbname</literal> in the
<link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>.
+ It's highly recommended that the said physical replication slot is named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ list on the primary, to prevent the subscriber from consuming changes
+ faster than the hot standby. Even when correctly configured, some latency
+ is expected when sending changes to logical subscribers due to the waiting
+ on slots named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>.
+ When <varname>standby_slot_names</varname> is utilized, the
+ primary server will not completely shut down until the corresponding
+ standbys, associated with the physical replication slots specified
+ in <varname>standby_slot_names</varname>, have confirmed
+ receiving the WAL up to the latest flushed position on the primary server.
</para>
<para>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index a1ff631e5e..b4dd5cce75 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -105,6 +105,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
MemoryContext per_query_ctx;
MemoryContext oldcontext;
XLogRecPtr end_of_wal;
+ XLogRecPtr wait_for_wal_lsn;
LogicalDecodingContext *ctx;
ResourceOwner old_resowner = CurrentResourceOwner;
ArrayType *arr;
@@ -224,6 +225,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
NameStr(MyReplicationSlot->data.plugin),
format_procedure(fcinfo->flinfo->fn_oid))));
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to wait_for_wal_lsn.
+ */
+ if (XLogRecPtrIsInvalid(upto_lsn))
+ wait_for_wal_lsn = end_of_wal;
+ else
+ wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
+
+ WaitForStandbyConfirmation(wait_for_wal_lsn);
+
ctx->output_writer_private = p;
/*
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index ad0fc6a04b..5074c8409f 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -488,6 +488,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
latestFlushPtr = GetStandbyFlushRecPtr(NULL);
if (remote_slot->confirmed_lsn > latestFlushPtr)
{
+ /*
+ * Can get here only if GUC 'standby_slot_names' on the primary server
+ * was not configured correctly.
+ */
ereport(AmLogicalSlotSyncWorkerProcess() ? LOG : ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("skipping slot synchronization as the received slot sync"
@@ -857,6 +861,13 @@ validate_remote_info(WalReceiverConn *wrconn)
remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull));
Assert(!isnull);
+ /*
+ * Slot sync is currently not supported on a cascading standby. This is
+ * because if we allow it, the primary server needs to wait for all the
+ * cascading standbys, otherwise, logical subscribers can still be ahead
+ * of one of the cascading standbys which we plan to promote. Thus, to
+ * avoid this additional complexity, we restrict it for the time being.
+ */
if (remote_in_recovery)
ereport(ERROR,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 2614f98ddd..dc463551ca 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -46,13 +46,17 @@
#include "common/string.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/interrupt.h"
#include "replication/slotsync.h"
#include "replication/slot.h"
+#include "replication/walsender_private.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
+#include "utils/varlena.h"
/*
* Replication slot on-disk data structure.
@@ -77,6 +81,21 @@ typedef struct ReplicationSlotOnDisk
ReplicationSlotPersistentData slotdata;
} ReplicationSlotOnDisk;
+/*
+ * Struct for the configuration of standby_slot_names.
+ *
+ * Note: this must be a flat representation that can be held in a single chunk
+ * of guc_malloc'd memory, so that it can be stored as the "extra" data for the
+ * standby_slot_names GUC.
+ */
+typedef struct
+{
+ int slot_num;
+
+ /* slot_names contains nmembers consecutive nul-terminated C strings */
+ char slot_names[FLEXIBLE_ARRAY_MEMBER];
+} StandbySlotConfigData;
+
/*
* Lookup table for slot invalidation causes.
*/
@@ -115,10 +134,25 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
/* My backend's replication slot in the shared memory array */
ReplicationSlot *MyReplicationSlot = NULL;
-/* GUC variable */
+/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+/*
+ * This GUC lists streaming replication standby server slot names that
+ * logical WAL sender processes will wait for.
+ */
+char *standby_slot_names;
+
+/* This is parsed and cached configuration for standby_slot_names */
+static StandbySlotConfigData *standby_slot_config;
+
+/*
+ * Oldest LSN that has been confirmed to be flushed to the standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
+
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
@@ -2345,3 +2379,374 @@ GetSlotInvalidationCause(const char *conflict_reason)
Assert(found);
return result;
}
+
+/*
+ * A helper function to validate slots specified in GUC standby_slot_names.
+ *
+ * The rawname will be parsed, and the parsed result will be saved into
+ * *elemlist.
+ */
+static bool
+validate_standby_slots(char *rawname, List **elemlist)
+{
+ bool ok;
+
+ /* Verify syntax and parse string into a list of identifiers */
+ ok = SplitIdentifierString(rawname, ',', elemlist);
+
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
+ else if (!ReplicationSlotCtl)
+ {
+ /*
+ * We cannot validate the replication slot if the replication slots'
+ * data has not been initialized. This is ok as we will anyway
+ * validate the specified slot when waiting for them to catch up. See
+ * StandbySlotsHaveCaughtup() for details.
+ */
+ }
+ else
+ {
+ /* Check that the specified slots exist and are logical slots */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, *elemlist)
+ {
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ GUC_check_errdetail("replication slot \"%s\" does not exist",
+ name);
+ ok = false;
+ break;
+ }
+
+ if (!SlotIsPhysical(slot))
+ {
+ GUC_check_errdetail("\"%s\" is not a physical replication slot",
+ name);
+ ok = false;
+ break;
+ }
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+ }
+
+ return ok;
+}
+
+/*
+ * GUC check_hook for standby_slot_names
+ */
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ char *rawname;
+ char *ptr;
+ List *elemlist;
+ int size;
+ StandbySlotConfigData *config;
+
+ if ((*newval)[0] == '\0')
+ return true;
+
+ /*
+ * "*" is not accepted as in that case the primary will not be able to
+ * know for which all standbys to wait for. It is inappropriate to block
+ * logical replication for physical slots that either lack associated
+ * standbys or have standbys associated that are not enabled for
+ * replication slot synchronization.
+ */
+ if (strcmp(*newval, "*") == 0)
+ {
+ GUC_check_errdetail("\"*\" is not accepted for standby_slot_names");
+ return false;
+ }
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Now verify if the specified slots exist and have correct type */
+ if (!validate_standby_slots(rawname, &elemlist))
+ {
+ pfree(rawname);
+ list_free(elemlist);
+ return false;
+ }
+
+ /* Compute the size required for the StandbySlotConfigData struct */
+ size = offsetof(StandbySlotConfigData, slot_names);
+ foreach_ptr(char, slot_name, elemlist)
+ size += strlen(slot_name) + 1;
+
+ /* GUC extra value must be guc_malloc'd, not palloc'd */
+ config = (StandbySlotConfigData *) guc_malloc(LOG, size);
+
+ /* Transform the data into StandbySlotConfigData */
+ config->slot_num = list_length(elemlist);
+
+ ptr = config->slot_names;
+ foreach_ptr(char, slot_name, elemlist)
+ {
+ strcpy(ptr, slot_name);
+ ptr += strlen(slot_name) + 1;
+ }
+
+ *extra = (void *) config;
+
+ pfree(rawname);
+ list_free(elemlist);
+ return true;
+}
+
+/*
+ * GUC assign_hook for standby_slot_names
+ */
+void
+assign_standby_slot_names(const char *newval, void *extra)
+{
+ /*
+ * The standby slots may have changed, so we must recompute the oldest
+ * LSN.
+ */
+ ss_oldest_flush_lsn = InvalidXLogRecPtr;
+
+ standby_slot_config = (StandbySlotConfigData *) extra;
+}
+
+/*
+ * Return true if the currently acquired slot is specified in
+ * standby_slot_names GUC; otherwise, return false.
+ */
+bool
+AcquiredStandbySlot(void)
+{
+ const char *name;
+
+ /* Return false if there is no value in standby_slot_names */
+ if (standby_slot_config == NULL)
+ return false;
+
+ name = standby_slot_config->slot_names;
+ for (int i = 0; i < standby_slot_config->slot_num; i++)
+ {
+ if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0)
+ return true;
+
+ name += strlen(name) + 1;
+ }
+
+ return false;
+}
+
+/*
+ * Return true if the slots specified in standby_slot_names have caught up to
+ * the given WAL location, false otherwise.
+ *
+ * The elevel parameter specifies the error level used for logging messages
+ * related to slots that do not exist, are invalidated, or are inactive.
+ */
+bool
+StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel)
+{
+ const char *name;
+ int caught_up_slot_num = 0;
+ XLogRecPtr min_restart_lsn = InvalidXLogRecPtr;
+
+ /*
+ * Don't need to wait for the standbys to catch up if there is no value in
+ * standby_slot_names.
+ */
+ if (standby_slot_config == NULL)
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if we are on a standby
+ * server, since we do not support syncing slots to cascading standbys.
+ */
+ if (RecoveryInProgress())
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if they are already
+ * beyond the specified WAL location.
+ */
+ if (!XLogRecPtrIsInvalid(ss_oldest_flush_lsn) &&
+ ss_oldest_flush_lsn >= wait_for_lsn)
+ return true;
+
+ /*
+ * To prevent concurrent slot dropping and creation while filtering the
+ * slots, take the ReplicationSlotControlLock outside of the loop.
+ */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ name = standby_slot_config->slot_names;
+ for (int i = 0; i < standby_slot_config->slot_num; i++)
+ {
+ XLogRecPtr restart_lsn;
+ bool invalidated;
+ bool inactive;
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ /*
+ * If a slot name provided in standby_slot_names does not exist,
+ * report a message and exit the loop. A user can specify a slot
+ * name that does not exist just before the server startup. The
+ * GUC check_hook(validate_standby_slots) cannot validate such a
+ * slot during startup as the ReplicationSlotCtl shared memory is
+ * not initialized at that time. It is also possible for a user to
+ * drop the slot in standby_slot_names afterwards.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not exist",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider creating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (SlotIsLogical(slot))
+ {
+ /*
+ * If a logical slot name is provided in standby_slot_names,
+ * report a message and exit the loop. Similar to the non-existent
+ * case, a user can specify a logical slot name in
+ * standby_slot_names before the server startup, or drop an
+ * existing physical slot and recreate a logical slot with the
+ * same name.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot have logical replication slot \"%s\" in parameter %s",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting for correction on \"%s\".",
+ name),
+ errhint("Consider removing logical slot \"%s\" from parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ SpinLockAcquire(&slot->mutex);
+ restart_lsn = slot->data.restart_lsn;
+ invalidated = slot->data.invalidated != RS_INVAL_NONE;
+ inactive = slot->active_pid == 0;
+ SpinLockRelease(&slot->mutex);
+
+ if (invalidated)
+ {
+ /* Specified physical slot has been invalidated */
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("physical slot \"%s\" specified in parameter %s has been invalidated",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider dropping and recreating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (XLogRecPtrIsInvalid(restart_lsn) || restart_lsn < wait_for_lsn)
+ {
+ /* Log a message if no active_pid for this physical slot */
+ if (inactive)
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not have active_pid",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider starting standby associated with \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+
+ /* Continue if the current slot hasn't caught up. */
+ break;
+ }
+
+ Assert(restart_lsn >= wait_for_lsn);
+
+ if (XLogRecPtrIsInvalid(min_restart_lsn) ||
+ min_restart_lsn > restart_lsn)
+ min_restart_lsn = restart_lsn;
+
+ caught_up_slot_num++;
+
+ name += strlen(name) + 1;
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /*
+ * Return false if not all the standbys have caught up to the specified
+ * WAL location.
+ */
+ if (caught_up_slot_num != standby_slot_config->slot_num)
+ return false;
+
+ /* The ss_oldest_flush_lsn must not retreat. */
+ Assert(XLogRecPtrIsInvalid(ss_oldest_flush_lsn) ||
+ min_restart_lsn >= ss_oldest_flush_lsn);
+
+ ss_oldest_flush_lsn = min_restart_lsn;
+
+ return true;
+}
+
+/*
+ * Wait for physical standbys to confirm receiving the given lsn.
+ *
+ * Used by logical decoding SQL functions. It waits for physical standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+void
+WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
+{
+ /*
+ * Don't need to wait for the standby to catch up if the current acquired
+ * slot is not a logical failover slot, or there is no value in
+ * standby_slot_names.
+ */
+ if (!MyReplicationSlot->data.failover || !standby_slot_config)
+ return;
+
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+
+ for (;;)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /* Exit if done waiting for every slot. */
+ if (StandbySlotsHaveCaughtup(wait_for_lsn, WARNING))
+ break;
+
+ /*
+ * Wait for the slots in the standby_slot_names to catch up, but use a
+ * timeout (1s) so we can also check if the standby_slot_names has
+ * been changed.
+ */
+ ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000,
+ WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION);
+ }
+
+ ConditionVariableCancelSleep();
+}
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 768a304723..ad79e1fccd 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -464,6 +464,12 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* crash, but this makes the data consistent after a clean shutdown.
*/
ReplicationSlotMarkDirty();
+
+ /*
+ * Wake up logical walsenders holding logical failover slots after
+ * updating the restart_lsn of the physical slot.
+ */
+ PhysicalWakeupLogicalWalSnd();
}
return retlsn;
@@ -504,6 +510,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto)
.segment_close = wal_segment_close),
NULL, NULL, NULL);
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to moveto lsn.
+ */
+ WaitForStandbyConfirmation(moveto);
+
/*
* Start reading at the slot's restart_lsn, which we know to point to
* a valid record.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 0f1047179c..d8bdc65dc1 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1725,26 +1725,111 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
ProcessPendingWrites();
}
+/*
+ * Wake up the logical walsender processes with logical failover slots if the
+ * currently acquired physical slot is specified in standby_slot_names GUC.
+ */
+void
+PhysicalWakeupLogicalWalSnd(void)
+{
+ Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot));
+
+ /*
+ * If we are running in a standby, there is no need to wake up walsenders.
+ * This is because we do not support syncing slots to cascading standbys,
+ * so, there are no walsenders waiting for standbys to catch up.
+ */
+ if (RecoveryInProgress())
+ return;
+
+ if (AcquiredStandbySlot())
+ ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv);
+}
+
+/*
+ * Returns true if not all standbys have caught up to the flushed position
+ * (flushed_lsn) when the current acquired slot is a logical failover
+ * slot and we are streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForStandbys(XLogRecPtr flushed_lsn, uint32 *wait_event)
+{
+ int elevel = got_STOPPING ? ERROR : WARNING;
+ bool failover_slot;
+
+ failover_slot = (replication_active && MyReplicationSlot->data.failover);
+
+ /*
+ * Note that after receiving the shutdown signal, an ERROR is reported if
+ * any slots are dropped, invalidated, or inactive. This measure is taken
+ * to prevent the walsender from waiting indefinitely.
+ */
+ if (failover_slot && !StandbySlotsHaveCaughtup(flushed_lsn, elevel))
+ {
+ *wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
+ return true;
+ }
+
+ *wait_event = 0;
+ return false;
+}
+
+/*
+ * Returns true if we need to wait for WALs to be flushed to disk, or if not
+ * all standbys have caught up to the flushed position (flushed_lsn) when the
+ * current acquired slot is a logical failover slot and we are
+ * streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
+{
+ /* Check if we need to wait for WALs to be flushed to disk */
+ if (target_lsn > flushed_lsn)
+ {
+ *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
+ return true;
+ }
+
+ /* Check if the standby slots have caught up to the flushed position */
+ return NeedToWaitForStandbys(flushed_lsn, wait_event);
+}
+
/*
* Wait till WAL < loc is flushed to disk so it can be safely sent to client.
*
- * Returns end LSN of flushed WAL. Normally this will be >= loc, but
- * if we detect a shutdown request (either from postmaster or client)
- * we will return early, so caller must always check.
+ * If the walsender holds a logical slot that has enabled failover, we also
+ * wait for all the specified streaming replication standby servers to confirm
+ * receipt of WAL up to RecentFlushPtr. It is beneficial to wait here for the
+ * confirmation up to RecentFlushPtr rather than waiting before transmitting
+ * each change to logical subscribers, which is already covered by
+ * RecentFlushPtr.
+ *
+ * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we
+ * detect a shutdown request (either from postmaster or client) we will return
+ * early, so caller must always check.
*/
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
int wakeEvents;
+ uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
- * have enough WAL available. This is particularly interesting if we're
- * far behind.
+ * have enough WAL available and all the standby servers have confirmed
+ * receipt of WAL up to RecentFlushPtr. This is particularly interesting
+ * if we're far behind.
*/
- if (RecentFlushPtr != InvalidXLogRecPtr &&
- loc <= RecentFlushPtr)
+ if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
return RecentFlushPtr;
/* Get a more recent flush pointer. */
@@ -1753,8 +1838,14 @@ WalSndWaitForWal(XLogRecPtr loc)
else
RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * Within the loop, we wait for the necessary WALs to be flushed to disk
+ * first, followed by waiting for standbys to catch up if there are enough
+ * WALs (see NeedToWaitForWal()) or upon receiving the shutdown signal.
+ */
for (;;)
{
+ bool wait_for_standby_at_stop = false;
long sleeptime;
/* Clear any already-pending wakeups */
@@ -1781,21 +1872,35 @@ WalSndWaitForWal(XLogRecPtr loc)
if (got_STOPPING)
XLogBackgroundFlush();
- /* Update our idea of the currently flushed position. */
- if (!RecoveryInProgress())
- RecentFlushPtr = GetFlushRecPtr(NULL);
- else
- RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * To avoid the scenario where standbys need to catch up to a newer
+ * WAL location in each iteration, we update our idea of the currently
+ * flushed position only if we are not waiting for standbys to catch
+ * up.
+ */
+ if (wait_event != WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ {
+ if (!RecoveryInProgress())
+ RecentFlushPtr = GetFlushRecPtr(NULL);
+ else
+ RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ }
/*
- * If postmaster asked us to stop, don't wait anymore.
+ * If postmaster asked us to stop and the standby slots have caught up
+ * to the flushed position, don't wait anymore.
*
* It's important to do this check after the recomputation of
* RecentFlushPtr, so we can send all remaining data before shutting
* down.
*/
if (got_STOPPING)
- break;
+ {
+ if (NeedToWaitForStandbys(RecentFlushPtr, &wait_event))
+ wait_for_standby_at_stop = true;
+ else
+ break;
+ }
/*
* We only send regular messages to the client for full decoded
@@ -1810,11 +1915,18 @@ WalSndWaitForWal(XLogRecPtr loc)
!waiting_for_ping_response)
WalSndKeepalive(false, InvalidXLogRecPtr);
- /* check whether we're done */
- if (loc <= RecentFlushPtr)
+ /*
+ * Exit the loop if already caught up and doesn't need to wait for
+ * standby slots.
+ */
+ if (!wait_for_standby_at_stop &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
break;
- /* Waiting for new WAL. Since we need to wait, we're now caught up. */
+ /*
+ * Waiting for new WAL or waiting for standbys to catch up. Since we
+ * need to wait, we're now caught up.
+ */
WalSndCaughtUp = true;
/*
@@ -1852,7 +1964,9 @@ WalSndWaitForWal(XLogRecPtr loc)
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL);
+ Assert(wait_event != 0);
+
+ WalSndWait(wakeEvents, sleeptime, wait_event);
}
/* reactivate latch so WalSndLoop knows to continue */
@@ -2262,6 +2376,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn)
{
ReplicationSlotMarkDirty();
ReplicationSlotsComputeRequiredLSN();
+ PhysicalWakeupLogicalWalSnd();
}
/*
@@ -3535,6 +3650,7 @@ WalSndShmemInit(void)
ConditionVariableInit(&WalSndCtl->wal_flush_cv);
ConditionVariableInit(&WalSndCtl->wal_replay_cv);
+ ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv);
}
}
@@ -3604,8 +3720,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event)
*
* And, we use separate shared memory CVs for physical and logical
* walsenders for selective wake ups, see WalSndWakeup() for more details.
+ *
+ * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV
+ * until awakened by physical walsenders after the walreceiver confirms
+ * the receipt of the LSN.
*/
- if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
+ if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+ else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv);
else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index ec2f31f82a..c08e00d1d6 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP
LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server."
LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server."
SSL_OPEN_SERVER "Waiting for SSL while attempting connection."
+WAIT_FOR_STANDBY_CONFIRMATION "Waiting for WAL to be received and flushed by the physical standby."
WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process."
WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 45013582a7..d77214795d 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4670,6 +4670,20 @@ struct config_string ConfigureNamesString[] =
check_debug_io_direct, assign_debug_io_direct, NULL
},
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("Lists streaming replication standby server slot "
+ "names that logical WAL sender processes will wait for."),
+ gettext_noop("Logical WAL sender processes will send decoded "
+ "changes to plugins only after the specified "
+ "replication slots confirm receiving WAL."),
+ GUC_LIST_INPUT
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, assign_standby_slot_names, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index edcc0282b2..2244ee52f7 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -343,6 +343,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsender processes will wait for
# - Standby Servers -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index acbf567150..73f7154173 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -226,6 +226,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
+extern PGDLLIMPORT char *standby_slot_names;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -274,4 +275,8 @@ extern void CheckSlotPermissions(void);
extern ReplicationSlotInvalidationCause
GetSlotInvalidationCause(const char *conflict_reason);
+extern bool AcquiredStandbySlot(void);
+extern bool StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel);
+extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn);
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index 0c3996e926..f2d8297f01 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -39,6 +39,7 @@ extern void InitWalSender(void);
extern bool exec_replication_command(const char *cmd_string);
extern void WalSndErrorCleanup(void);
extern void WalSndResourceCleanup(bool isCommit);
+extern void PhysicalWakeupLogicalWalSnd(void);
extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
extern void WalSndSignals(void);
extern Size WalSndShmemSize(void);
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 3113e9ea47..109924ffcd 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -113,6 +113,13 @@ typedef struct
ConditionVariable wal_flush_cv;
ConditionVariable wal_replay_cv;
+ /*
+ * Used by physical walsenders holding slots specified in
+ * standby_slot_names to wake up logical walsenders holding logical
+ * failover slots when a walreceiver confirms the receipt of LSN.
+ */
+ ConditionVariable wal_confirm_rcv_cv;
+
WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER];
} WalSndCtlData;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index c8a7aa9a11..d64dc5fcdb 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra,
extern void assign_wal_consistency_checking(const char *newval, void *extra);
extern bool check_wal_segment_size(int *newval, void **extra, GucSource source);
extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
+extern bool check_standby_slot_names(char **newval, void **extra,
+ GucSource source);
+extern void assign_standby_slot_names(const char *newval, void *extra);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl
index 5c7b4ca5e3..b95d95c06f 100644
--- a/src/test/recovery/t/006_logical_decoding.pl
+++ b/src/test/recovery/t/006_logical_decoding.pl
@@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'},
undef, 'logical slot was actually dropped with DB');
# Test logical slot advancing and its durability.
+# Passing failover=true (last arg) should not have any impact on advancing.
my $logical_slot = 'logical_slot';
$node_primary->safe_psql('postgres',
- "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);"
+ "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);"
);
$node_primary->psql(
'postgres', "
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 021c58f621..99025b352a 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -504,18 +504,258 @@ ok( $standby1->poll_query_until(
"SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot' AND synced AND NOT temporary;"),
'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby');
+##################################################
+# Test that logical failover replication slots wait for the specified
+# physical replication slots to receive the changes first. It uses the
+# following set up:
+#
+# (physical standbys)
+# | ----> standby1 (primary_slot_name = sb1_slot)
+# | ----> standby2 (primary_slot_name = sb2_slot)
+# primary ----- |
+# (logical replication)
+# | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
+# | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
+#
+# standby_slot_names = 'sb1_slot'
+#
+# The setup is configured in such a way that the logical slot of subscriber1 is
+# enabled for failover, and thus the subscriber1 will wait for the physical
+# slot of standby1(sb1_slot) to catch up before receiving the decoded changes.
+##################################################
+
+$backup_name = 'backup3';
+
+$primary->psql('postgres',
+ q{SELECT pg_create_physical_replication_slot('sb2_slot');});
+
+$primary->backup($backup_name);
+
+# Create another standby
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup(
+ $primary, $backup_name,
+ has_streaming => 1,
+ has_restoring => 1);
+$standby2->append_conf(
+ 'postgresql.conf', qq(
+primary_slot_name = 'sb2_slot'
+));
+$standby2->start;
+$primary->wait_for_replay_catchup($standby2);
+
+# Configure primary to disallow any logical slots that have enabled failover
+# from getting ahead of the specified physical replication slot (sb1_slot).
+$primary->append_conf(
+ 'postgresql.conf', qq(
+standby_slot_names = 'sb1_slot'
+));
+$primary->reload;
+
+# Create another subscriber node without enabling failover, wait for sync to
+# complete
+my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$subscriber2->init;
+$subscriber2->start;
+$subscriber2->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE tab_int (a int PRIMARY KEY);
+ CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot);
+]);
+
+$subscriber2->wait_for_subscription_sync;
+
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+my $offset = -s $primary->logfile;
+
+# Stop the standby associated with the specified physical replication slot
+# (sb1_slot) so that the logical replication slot (lsub1_slot) won't receive
+# changes until the standby comes up.
+$standby1->stop;
+
+# Create some data on the primary
+my $primary_row_count = 20;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(11, $primary_row_count);");
+
+# Wait until the standby2 that's still running gets the data from the primary
+$primary->wait_for_replay_catchup($standby2);
+$result = $standby2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby2 gets data from primary");
+
+# Wait for regress_mysub2 to get the data from the primary. This subscription
+# was not enabled for failover so it gets the data without waiting for any
+# standbys.
+$primary->wait_for_catchup('regress_mysub2');
+$result = $subscriber2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "subscriber2 gets data from primary");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# The regress_mysub1 was enabled for failover so it doesn't get the data from
+# primary and keeps waiting for the standby specified in standby_slot_names
+# (sb1_slot aka standby1).
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) <> $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data from primary until standby1 acknowledges changes"
+);
+
+# Start the standby specified in standby_slot_names (sb1_slot aka standby1) and
+# wait for it to catch up with the primary.
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+$result = $standby1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby1 gets data from primary");
+
+# Now that the standby specified in standby_slot_names is up and running, the
+# primary can send the decoded changes to the subscription enabled for failover
+# (i.e. regress_mysub1). While the standby was down, regress_mysub1 didn't
+# receive any data from the primary. i.e. the primary didn't allow it to go
+# ahead of standby.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 acknowledges changes");
+
+##################################################
+# Verify that when using pg_logical_slot_get_changes to consume changes from a
+# logical slot with failover enabled, it will also wait for the slots specified
+# in standby_slot_names to catch up.
+##################################################
+
+# Stop the standby associated with the specified physical replication slot so
+# that the logical replication slot won't receive changes until the standby
+# slot's restart_lsn is advanced or the slot is removed from the
+# standby_slot_names list.
+$primary->safe_psql('postgres', "TRUNCATE tab_int;");
+$primary->wait_for_catchup('regress_mysub1');
+$standby1->stop;
+
+# Disable the regress_mysub1 to prevent the logical walsender from generating
+# more warnings.
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 DISABLE");
+
+# Wait for the replication slot to become inactive on the publisher
+$primary->poll_query_until(
+ 'postgres',
+ "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'",
+ 1);
+
+# Create a logical 'test_decoding' replication slot with failover enabled
+$primary->safe_psql('postgres',
+ "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);"
+);
+
+my $back_q = $primary->background_psql(
+ 'postgres',
+ on_error_stop => 0,
+ timeout => $PostgreSQL::Test::Utils::timeout_default);
+
+# pg_logical_slot_get_changes will be blocked until the standby catches up,
+# hence it needs to be executed in a background session.
+$offset = -s $primary->logfile;
+$back_q->query_until(
+ qr/logical_slot_get_changes/, q(
+ \echo logical_slot_get_changes
+ SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);
+));
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the function
+# pg_logical_slot_get_changes should now return, and the session can be
+# stopped.
+$back_q->quit;
+
+$primary->safe_psql('postgres',
+ "SELECT pg_drop_replication_slot('test_slot');"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
+# Enable the regress_mysub1 for further tests
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+##################################################
+# Test that logical replication will wait for the user-created inactive
+# physical slot to catch up until we remove the slot from standby_slot_names.
+##################################################
+
+$offset = -s $primary->logfile;
+
+# Create some data on the primary
+$primary_row_count = 10;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# The regress_mysub1 doesn't get the data from primary because the specified
+# standby slot (sb1_slot) in standby_slot_names is inactive.
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data as the sb1_slot doesn't catch up");
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the primary server should now
+# send the decoded changes to the subscription.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
##################################################
# Promote the standby1 to primary. Confirm that:
# a) the slot 'lsub1_slot' is retained on the new primary
# b) logical replication for regress_mysub1 is resumed successfully after failover
##################################################
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+
$standby1->promote;
# Update subscription with the new primary's connection info
my $standby1_conninfo = $standby1->connstr . ' dbname=postgres';
$subscriber1->safe_psql('postgres',
- "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo';
- ALTER SUBSCRIPTION regress_mysub1 ENABLE; ");
+ "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo';");
# Confirm the synced slot 'lsub1_slot' is retained on the new primary
is($standby1->safe_psql('postgres',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 95ae7845d8..c0b9875838 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2650,6 +2650,7 @@ SplitPoint
SplitTextOutputData
SplitVar
StackElem
+StandbySlotConfigData
StartDataPtrType
StartLOPtrType
StartLOsPtrType
--
2.34.1
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-07 03:19 Masahiko Sawada <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 0 replies; 68+ messages in thread
From: Masahiko Sawada @ 2024-03-07 03:19 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Peter Smith <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Wed, Mar 6, 2024 at 5:53 PM Amit Kapila <[email protected]> wrote:
>
> On Wed, Mar 6, 2024 at 12:07 PM Masahiko Sawada <[email protected]> wrote:
> >
> > On Fri, Mar 1, 2024 at 3:22 PM Peter Smith <[email protected]> wrote:
> > >
> > > On Fri, Mar 1, 2024 at 5:11 PM Masahiko Sawada <[email protected]> wrote:
> > > >
> > > ...
> > > > + /*
> > > > + * "*" is not accepted as in that case primary will not be able to know
> > > > + * for which all standbys to wait for. Even if we have physical slots
> > > > + * info, there is no way to confirm whether there is any standby
> > > > + * configured for the known physical slots.
> > > > + */
> > > > + if (strcmp(*newval, "*") == 0)
> > > > + {
> > > > + GUC_check_errdetail("\"*\" is not accepted for
> > > > standby_slot_names");
> > > > + return false;
> > > > + }
> > > >
> > > > Why only '*' is checked aside from validate_standby_slots()? I think
> > > > that the doc doesn't mention anything about '*' and '*' cannot be used
> > > > as a replication slot name. So even if we don't have this check, it
> > > > might be no problem.
> > > >
> > >
> > > Hi, a while ago I asked this same question. See [1 #28] for the response..
> >
> > Thanks. Quoting the response from the email:
> >
> > SplitIdentifierString() does not give error for '*' and '*' can be considered
> > as valid value which if accepted can mislead user that all the standbys's slots
> > are now considered, which is not the case here. So we want to explicitly call
> > out this case i.e. '*' is not accepted as valid value for standby_slot_names.
> >
> > IIUC we're concerned with a case like where the user confused
> > standby_slot_names values with synchronous_standby_names values. Which
> > means we would need to keep thath check consistent with available
> > values of synchronous_standby_names.
> >
>
> Both have different formats to specify. For example, for
> synchronous_standby_names we have the following kind of syntax to
> specify:
> [FIRST] num_sync ( standby_name [, ...] )
> ANY num_sync ( standby_name [, ...] )
> standby_name [, ...]
>
> I don't think we can have a common check for both of them as the
> specifications are different. In fact, I don't think we need a special
> check for '*'.
I think so too.
> The user will anyway get a WARNING at a later point
> that the replication slot with that name doesn't exist.
Right.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-07 04:46 Amit Kapila <[email protected]>
parent: Peter Smith <[email protected]>
1 sibling, 1 reply; 68+ messages in thread
From: Amit Kapila @ 2024-03-07 04:46 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Thu, Mar 7, 2024 at 7:35 AM Peter Smith <[email protected]> wrote:
>
> Here are some review comments for v107-0001
>
> ======
> src/backend/replication/slot.c
>
> 1.
> +/*
> + * Struct for the configuration of standby_slot_names.
> + *
> + * Note: this must be a flat representation that can be held in a single chunk
> + * of guc_malloc'd memory, so that it can be stored as the "extra" data for the
> + * standby_slot_names GUC.
> + */
> +typedef struct
> +{
> + int slot_num;
> +
> + /* slot_names contains nmembers consecutive nul-terminated C strings */
> + char slot_names[FLEXIBLE_ARRAY_MEMBER];
> +} StandbySlotConfigData;
> +
>
> 1a.
> To avoid any ambiguity this 1st field is somehow a slot ID number, I
> felt a better name would be 'nslotnames' or even just 'n' or 'count',
>
We can probably just add a comment above slot_num and that should be
sufficient but I am fine with 'nslotnames' as well, in anycase let's
add a comment for the same.
>
> 6b.
> IMO this function would be tidier written such that the
> MyReplicationSlot->data.name is passed as a parameter. Then you can
> name the function more naturally like:
>
> IsSlotInStandbySlotNames(const char *slot_name)
>
+1. How about naming it as SlotExistsinStandbySlotNames(char
*slot_name) and pass the slot_name from MyReplicationSlot? Otherwise,
we need an Assert for MyReplicationSlot in this function.
Also, can we add a comment like below before the loop:
+ /*
+ * XXX: We are not expecting this list to be long so a linear search
+ * shouldn't hurt but if that turns out not to be true then we can cache
+ * this information for each WalSender as well.
+ */
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-07 05:54 Amit Kapila <[email protected]>
parent: shveta malik <[email protected]>
0 siblings, 0 replies; 68+ messages in thread
From: Amit Kapila @ 2024-03-07 05:54 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; Masahiko Sawada <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Thu, Mar 7, 2024 at 8:37 AM shveta malik <[email protected]> wrote:
>
I thought about whether we can make standby_slot_names as USERSET
instead of SIGHUP and it doesn't sound like a good idea as that can
lead to inconsistent standby replicas even after configuring the
correct value of standby_slot_names. One can set a different or ''
(empty) value for a particular session and consume all changes from
the slot without waiting for the standby to acknowledge the change.
Also, it would be difficult for users to ensure whether the standby is
always ahead of subscribers. Does anyone think differently?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-07 06:28 Zhijie Hou (Fujitsu) <[email protected]>
parent: Peter Smith <[email protected]>
1 sibling, 0 replies; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-07 06:28 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Amit Kapila <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Thursday, March 7, 2024 10:05 AM Peter Smith <[email protected]> wrote:
>
> Here are some review comments for v107-0001
Thanks for the comments.
>
> ======
> src/backend/replication/slot.c
>
> 1.
> +/*
> + * Struct for the configuration of standby_slot_names.
> + *
> + * Note: this must be a flat representation that can be held in a
> +single chunk
> + * of guc_malloc'd memory, so that it can be stored as the "extra" data
> +for the
> + * standby_slot_names GUC.
> + */
> +typedef struct
> +{
> + int slot_num;
> +
> + /* slot_names contains nmembers consecutive nul-terminated C strings
> +*/ char slot_names[FLEXIBLE_ARRAY_MEMBER];
> +} StandbySlotConfigData;
> +
>
> 1a.
> To avoid any ambiguity this 1st field is somehow a slot ID number, I felt a better
> name would be 'nslotnames' or even just 'n' or 'count',
Changed to 'nslotnames'.
>
> ~
>
> 1b.
> (fix typo)
>
> SUGGESTION for the 2nd field comment
> slot_names is a chunk of 'n' X consecutive null-terminated C strings
Changed.
>
> ~
>
> 1c.
> A more explanatory name for this typedef maybe is
> 'StandbySlotNamesConfigData' ?
Changed.
>
> ~~~
>
>
> 2.
> +/* This is parsed and cached configuration for standby_slot_names */
> +static StandbySlotConfigData *standby_slot_config;
>
>
> 2a.
> /This is parsed and cached configuration for .../This is the parsed and cached
> configuration for .../
Changed.
>
> ~
>
> 2b.
> Similar to above -- since this only has name information maybe it is more
> correct to call it 'standby_slot_names_config'?
>
Changed.
> ~~~
>
> 3.
> +/*
> + * A helper function to validate slots specified in GUC standby_slot_names.
> + *
> + * The rawname will be parsed, and the parsed result will be saved into
> + * *elemlist.
> + */
> +static bool
> +validate_standby_slots(char *rawname, List **elemlist)
>
> /and the parsed result/and the result/
>
Changed.
> ~~~
>
> 4. check_standby_slot_names
>
> + /* Need a modifiable copy of string */ rawname = pstrdup(*newval);
>
> /copy of string/copy of the GUC string/
>
Changed.
> ~~~
>
> 5.
> +assign_standby_slot_names(const char *newval, void *extra) {
> + /*
> + * The standby slots may have changed, so we must recompute the oldest
> + * LSN.
> + */
> + ss_oldest_flush_lsn = InvalidXLogRecPtr;
> +
> + standby_slot_config = (StandbySlotConfigData *) extra; }
>
> To avoid leaking don't we need to somewhere take care to free any memory
> used by a previous value (if any) of this 'standby_slot_config'?
>
The memory of extra is maintained by the GUC mechanism. It will be
automatically freed when the associated GUC setting is no longer of interest.
See src/backend/utils/misc/README for details.
> ~~~
>
> 6. AcquiredStandbySlot
>
> +/*
> + * Return true if the currently acquired slot is specified in
> + * standby_slot_names GUC; otherwise, return false.
> + */
> +bool
> +AcquiredStandbySlot(void)
> +{
> + const char *name;
> +
> + /* Return false if there is no value in standby_slot_names */ if
> + (standby_slot_config == NULL) return false;
> +
> + name = standby_slot_config->slot_names; for (int i = 0; i <
> + standby_slot_config->slot_num; i++) { if (strcmp(name,
> + NameStr(MyReplicationSlot->data.name)) == 0) return true;
> +
> + name += strlen(name) + 1;
> + }
> +
> + return false;
> +}
>
> 6a.
> Just checking "(standby_slot_config == NULL)" doesn't seem enough to me,
> because IIUIC it is possible when 'standby_slot_names' has no value then
> maybe standby_slot_config is not NULL but standby_slot_config->slot_num is
> 0.
The standby_slot_config will always be NULL if there is no value in it.
While checking, I did find a rare case that if there are only some white space
in the standby_slot_names, then slot_num will be 0, and have fixed it so that
standby_slot_config will always be NULL if there is no meaning value in guc.
>
> ~
>
> 6b.
> IMO this function would be tidier written such that the
> MyReplicationSlot->data.name is passed as a parameter. Then you can
> name the function more naturally like:
>
> IsSlotInStandbySlotNames(const char *slot_name)
Changed it to SlotExistsInStandbySlotNames.
>
> ~
>
> 6c.
> IMO the body of the function will be tidier if written so there are only 2 returns
> instead of 3 like
>
> SUGGESTION:
> if (...)
> {
> for (...)
> {
> ...
> return true;
> }
> }
> return false;
I personally prefer the current style.
>
> ~~~
>
> 7.
> + /*
> + * Don't need to wait for the standbys to catch up if there is no value
> + in
> + * standby_slot_names.
> + */
> + if (standby_slot_config == NULL)
> + return true;
>
> (similar to a previous review comment)
>
> This check doesn't seem enough because IIUIC it is possible when
> 'standby_slot_names' has no value then maybe standby_slot_config is not NULL
> but standby_slot_config->slot_num is 0.
Same as above.
>
> ~~~
>
> 8. WaitForStandbyConfirmation
>
> + /*
> + * Don't need to wait for the standby to catch up if the current
> + acquired
> + * slot is not a logical failover slot, or there is no value in
> + * standby_slot_names.
> + */
> + if (!MyReplicationSlot->data.failover || !standby_slot_config) return;
>
> (similar to a previous review comment)
>
> IIUIC it is possible that when 'standby_slot_names' has no value, then
> standby_slot_config is not NULL but standby_slot_config->slot_num is 0. So
> shouldn't that be checked too?
>
> Perhaps it is convenient to encapsulate this check using some macro:
> #define StandbySlotNamesHasNoValue() (standby_slot_config = NULL ||
> standby_slot_config->slot_num == 0)
Same as above, I think we can avoid checking slot_num.
Best Regards,
Hou zj
^ permalink raw reply [nested|flat] 68+ messages in thread
* RE: Synchronizing slots from primary to standby
@ 2024-03-07 06:30 Zhijie Hou (Fujitsu) <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-03-07 06:30 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; Peter Smith <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Thursday, March 7, 2024 12:46 PM Amit Kapila <[email protected]> wrote:
>
> On Thu, Mar 7, 2024 at 7:35 AM Peter Smith <[email protected]>
> wrote:
> >
> > Here are some review comments for v107-0001
> >
> > ======
> > src/backend/replication/slot.c
> >
> > 1.
> > +/*
> > + * Struct for the configuration of standby_slot_names.
> > + *
> > + * Note: this must be a flat representation that can be held in a single chunk
> > + * of guc_malloc'd memory, so that it can be stored as the "extra" data for
> the
> > + * standby_slot_names GUC.
> > + */
> > +typedef struct
> > +{
> > + int slot_num;
> > +
> > + /* slot_names contains nmembers consecutive nul-terminated C strings */
> > + char slot_names[FLEXIBLE_ARRAY_MEMBER];
> > +} StandbySlotConfigData;
> > +
> >
> > 1a.
> > To avoid any ambiguity this 1st field is somehow a slot ID number, I
> > felt a better name would be 'nslotnames' or even just 'n' or 'count',
> >
>
> We can probably just add a comment above slot_num and that should be
> sufficient but I am fine with 'nslotnames' as well, in anycase let's
> add a comment for the same.
Added.
>
> >
> > 6b.
> > IMO this function would be tidier written such that the
> > MyReplicationSlot->data.name is passed as a parameter. Then you can
> > name the function more naturally like:
> >
> > IsSlotInStandbySlotNames(const char *slot_name)
> >
>
> +1. How about naming it as SlotExistsinStandbySlotNames(char
> *slot_name) and pass the slot_name from MyReplicationSlot? Otherwise,
> we need an Assert for MyReplicationSlot in this function.
Changed as suggested.
>
> Also, can we add a comment like below before the loop:
> + /*
> + * XXX: We are not expecting this list to be long so a linear search
> + * shouldn't hurt but if that turns out not to be true then we can cache
> + * this information for each WalSender as well.
> + */
Added.
Attach the V108 patch set which addressed above and Peter's comments.
I also removed the check for "*" in guc check hook.
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v108-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch (49.8K, ../../OS0PR01MB57168AB61BF1EF3F1413153C94202@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v108-0001-Allow-logical-walsenders-to-wait-for-the-physic.patch)
download | inline diff:
From a602dd613ebd231f11b08c31be7e2e5de3cbba58 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Thu, 7 Mar 2024 13:46:32 +0800
Subject: [PATCH v108] Allow logical walsenders to wait for the physical
standby
This patch introduces a mechanism to ensure that physical standby servers,
which are potential failover candidates, have received and flushed changes
before making them visible to subscribers. By doing so, it guarantees that
the promoted standby server is not lagging behind the subscribers when a
failover is necessary.
A new parameter named 'standby_slot_names' is introduced. The logical
walsender now guarantees that all local changes are sent and flushed to
the standby servers corresponding to the replication slots specified in
'standby_slot_names' before sending those changes to the subscriber.
Additionally, The SQL functions pg_logical_slot_get_changes,
pg_logical_slot_peek_changes and pg_replication_slot_advance are modified.
Now, when used with logical failover slots, these functions will
block until all physical slots specified in 'standby_slot_names' have
confirmed WAL receipt.
---
doc/src/sgml/config.sgml | 39 ++
doc/src/sgml/func.sgml | 16 +-
doc/src/sgml/logicaldecoding.sgml | 12 +
.../replication/logical/logicalfuncs.c | 12 +
src/backend/replication/logical/slotsync.c | 11 +
src/backend/replication/slot.c | 405 +++++++++++++++++-
src/backend/replication/slotfuncs.c | 12 +
src/backend/replication/walsender.c | 160 ++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 14 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/replication/slot.h | 5 +
src/include/replication/walsender.h | 1 +
src/include/replication/walsender_private.h | 7 +
src/include/utils/guc_hooks.h | 3 +
src/test/recovery/t/006_logical_decoding.pl | 3 +-
.../t/040_standby_failover_slots_sync.pl | 244 ++++++++++-
src/tools/pgindent/typedefs.list | 1 +
18 files changed, 921 insertions(+), 27 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index b38cbd714a..7f821bc4e3 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4559,6 +4559,45 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Lists the streaming replication standby server slot names that logical
+ WAL sender processes will wait for. Logical WAL sender processes will
+ send decoded changes to plugins only after the specified replication
+ slots confirm receiving WAL. This guarantees that logical replication
+ slots with failover enabled do not consume changes until those changes
+ are received and flushed to corresponding physical standbys. If a
+ logical replication connection is meant to switch to a physical standby
+ after the standby is promoted, the physical replication slot for the
+ standby should be listed here. Note that logical replication will not
+ proceed if the slots specified in the
+ <varname>standby_slot_names</varname> do not exist or are invalidated.
+ Additionally, the replication management functions
+ <link linkend="pg-replication-slot-advance">
+ <function>pg_replication_slot_advance</function></link>,
+ <link linkend="pg-logical-slot-get-changes">
+ <function>pg_logical_slot_get_changes</function></link>, and
+ <link linkend="pg-logical-slot-peek-changes">
+ <function>pg_logical_slot_peek_changes</function></link>,
+ when used with logical failover slots, will block until all
+ physical slots specified in <varname>standby_slot_names</varname> have
+ confirmed WAL receipt.
+ </para>
+ <para>
+ The standbys corresponding to the physical replication slots in
+ <varname>standby_slot_names</varname> must configure
+ <literal>sync_replication_slots = true</literal> so they can receive
+ logical failover slot changes from the primary.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index e5fa82c161..0bb7aeb40e 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -28150,7 +28150,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-get-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_get_changes</primary>
</indexterm>
@@ -28173,11 +28173,15 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
the specified value. Note, however, that the actual number of
rows returned may be larger, since this limit is only checked after
adding the rows produced when decoding each new transaction commit.
+ If the specified slot is a logical failover slot then the function will
+ not return until all physical slots specified in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ have confirmed WAL receipt.
</para></entry>
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-logical-slot-peek-changes" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_logical_slot_peek_changes</primary>
</indexterm>
@@ -28232,7 +28236,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
</row>
<row>
- <entry role="func_table_entry"><para role="func_signature">
+ <entry id="pg-replication-slot-advance" role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_replication_slot_advance</primary>
</indexterm>
@@ -28248,7 +28252,11 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset
the name of the slot and the actual position that it was advanced to.
The updated slot position information is written out at the next
checkpoint if any advancing is done. So in the event of a crash, the
- slot may return to an earlier position.
+ slot may return to an earlier position. If the specified slot is a
+ logical failover slot then the function will not return until all
+ physical slots specified in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ have confirmed WAL receipt.
</para></entry>
</row>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8f13780e74..5a15bbc580 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -384,6 +384,18 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU
must be enabled on the standby. It is also necessary to specify a valid
<literal>dbname</literal> in the
<link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>.
+ It's highly recommended that the said physical replication slot is named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>
+ list on the primary, to prevent the subscriber from consuming changes
+ faster than the hot standby. Even when correctly configured, some latency
+ is expected when sending changes to logical subscribers due to the waiting
+ on slots named in
+ <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link>.
+ When <varname>standby_slot_names</varname> is utilized, the
+ primary server will not completely shut down until the corresponding
+ standbys, associated with the physical replication slots specified
+ in <varname>standby_slot_names</varname>, have confirmed
+ receiving the WAL up to the latest flushed position on the primary server.
</para>
<para>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index a1ff631e5e..b4dd5cce75 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -105,6 +105,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
MemoryContext per_query_ctx;
MemoryContext oldcontext;
XLogRecPtr end_of_wal;
+ XLogRecPtr wait_for_wal_lsn;
LogicalDecodingContext *ctx;
ResourceOwner old_resowner = CurrentResourceOwner;
ArrayType *arr;
@@ -224,6 +225,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
NameStr(MyReplicationSlot->data.plugin),
format_procedure(fcinfo->flinfo->fn_oid))));
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to wait_for_wal_lsn.
+ */
+ if (XLogRecPtrIsInvalid(upto_lsn))
+ wait_for_wal_lsn = end_of_wal;
+ else
+ wait_for_wal_lsn = Min(upto_lsn, end_of_wal);
+
+ WaitForStandbyConfirmation(wait_for_wal_lsn);
+
ctx->output_writer_private = p;
/*
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index ad0fc6a04b..5074c8409f 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -488,6 +488,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
latestFlushPtr = GetStandbyFlushRecPtr(NULL);
if (remote_slot->confirmed_lsn > latestFlushPtr)
{
+ /*
+ * Can get here only if GUC 'standby_slot_names' on the primary server
+ * was not configured correctly.
+ */
ereport(AmLogicalSlotSyncWorkerProcess() ? LOG : ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("skipping slot synchronization as the received slot sync"
@@ -857,6 +861,13 @@ validate_remote_info(WalReceiverConn *wrconn)
remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull));
Assert(!isnull);
+ /*
+ * Slot sync is currently not supported on a cascading standby. This is
+ * because if we allow it, the primary server needs to wait for all the
+ * cascading standbys, otherwise, logical subscribers can still be ahead
+ * of one of the cascading standbys which we plan to promote. Thus, to
+ * avoid this additional complexity, we restrict it for the time being.
+ */
if (remote_in_recovery)
ereport(ERROR,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 2614f98ddd..aa83830569 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -46,13 +46,17 @@
#include "common/string.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/interrupt.h"
#include "replication/slotsync.h"
#include "replication/slot.h"
+#include "replication/walsender_private.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
+#include "utils/varlena.h"
/*
* Replication slot on-disk data structure.
@@ -77,6 +81,25 @@ typedef struct ReplicationSlotOnDisk
ReplicationSlotPersistentData slotdata;
} ReplicationSlotOnDisk;
+/*
+ * Struct for the configuration of standby_slot_names.
+ *
+ * Note: this must be a flat representation that can be held in a single chunk
+ * of guc_malloc'd memory, so that it can be stored as the "extra" data for the
+ * standby_slot_names GUC.
+ */
+typedef struct
+{
+ /* Number of slot names in the slot_names[] */
+ int nslotnames;
+
+ /*
+ * slot_names is a chunk of 'nslotnames' consecutive null-terminated C
+ * strings.
+ */
+ char slot_names[FLEXIBLE_ARRAY_MEMBER];
+} StandbySlotNamesConfigData;
+
/*
* Lookup table for slot invalidation causes.
*/
@@ -115,10 +138,25 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
/* My backend's replication slot in the shared memory array */
ReplicationSlot *MyReplicationSlot = NULL;
-/* GUC variable */
+/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+/*
+ * This GUC lists streaming replication standby server slot names that
+ * logical WAL sender processes will wait for.
+ */
+char *standby_slot_names;
+
+/* This is the parsed and cached configuration for standby_slot_names */
+static StandbySlotNamesConfigData *standby_slot_names_config;
+
+/*
+ * Oldest LSN that has been confirmed to be flushed to the standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
+
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
@@ -2345,3 +2383,368 @@ GetSlotInvalidationCause(const char *conflict_reason)
Assert(found);
return result;
}
+
+/*
+ * A helper function to validate slots specified in GUC standby_slot_names.
+ *
+ * The rawname will be parsed, and the result will be saved into *elemlist.
+ */
+static bool
+validate_standby_slots(char *rawname, List **elemlist)
+{
+ bool ok;
+
+ /* Verify syntax and parse string into a list of identifiers */
+ ok = SplitIdentifierString(rawname, ',', elemlist);
+
+ if (!ok)
+ {
+ GUC_check_errdetail("List syntax is invalid.");
+ }
+ else if (!ReplicationSlotCtl)
+ {
+ /*
+ * We cannot validate the replication slot if the replication slots'
+ * data has not been initialized. This is ok as we will anyway
+ * validate the specified slot when waiting for them to catch up. See
+ * StandbySlotsHaveCaughtup() for details.
+ */
+ }
+ else
+ {
+ /* Check that the specified slots exist and are logical slots */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ foreach_ptr(char, name, *elemlist)
+ {
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ GUC_check_errdetail("replication slot \"%s\" does not exist",
+ name);
+ ok = false;
+ break;
+ }
+
+ if (!SlotIsPhysical(slot))
+ {
+ GUC_check_errdetail("\"%s\" is not a physical replication slot",
+ name);
+ ok = false;
+ break;
+ }
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+ }
+
+ return ok;
+}
+
+/*
+ * GUC check_hook for standby_slot_names
+ */
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ char *rawname;
+ char *ptr;
+ List *elemlist;
+ int size;
+ bool ok;
+ StandbySlotNamesConfigData *config;
+
+ if ((*newval)[0] == '\0')
+ return true;
+
+ /* Need a modifiable copy of the GUC string */
+ rawname = pstrdup(*newval);
+
+ /* Now verify if the specified slots exist and have correct type */
+ ok = validate_standby_slots(rawname, &elemlist);
+
+ if (!ok || elemlist == NIL)
+ {
+ pfree(rawname);
+ list_free(elemlist);
+ return ok;
+ }
+
+ /* Compute the size required for the StandbySlotNamesConfigData struct */
+ size = offsetof(StandbySlotNamesConfigData, slot_names);
+ foreach_ptr(char, slot_name, elemlist)
+ size += strlen(slot_name) + 1;
+
+ /* GUC extra value must be guc_malloc'd, not palloc'd */
+ config = (StandbySlotNamesConfigData *) guc_malloc(LOG, size);
+
+ /* Transform the data into StandbySlotNamesConfigData */
+ config->nslotnames = list_length(elemlist);
+
+ ptr = config->slot_names;
+ foreach_ptr(char, slot_name, elemlist)
+ {
+ strcpy(ptr, slot_name);
+ ptr += strlen(slot_name) + 1;
+ }
+
+ *extra = (void *) config;
+
+ pfree(rawname);
+ list_free(elemlist);
+ return true;
+}
+
+/*
+ * GUC assign_hook for standby_slot_names
+ */
+void
+assign_standby_slot_names(const char *newval, void *extra)
+{
+ /*
+ * The standby slots may have changed, so we must recompute the oldest
+ * LSN.
+ */
+ ss_oldest_flush_lsn = InvalidXLogRecPtr;
+
+ standby_slot_names_config = (StandbySlotNamesConfigData *) extra;
+}
+
+/*
+ * Check if the passed slot_name is specified in the standby_slot_names GUC.
+ * Returns true if the slot exists in the GUC; otherwise, returns false.
+ */
+bool
+SlotExistsInStandbySlotNames(const char *slot_name)
+{
+ const char *standby_slot_name;
+
+ /* Return false if there is no value in standby_slot_names */
+ if (standby_slot_names_config == NULL)
+ return false;
+
+ /*
+ * XXX: We are not expecting this list to be long so a linear search
+ * shouldn't hurt but if that turns out not to be true then we can cache
+ * this information for each WalSender as well.
+ */
+ standby_slot_name = standby_slot_names_config->slot_names;
+ for (int i = 0; i < standby_slot_names_config->nslotnames; i++)
+ {
+ if (strcmp(standby_slot_name, slot_name) == 0)
+ return true;
+
+ standby_slot_name += strlen(standby_slot_name) + 1;
+ }
+
+ return false;
+}
+
+/*
+ * Return true if the slots specified in standby_slot_names have caught up to
+ * the given WAL location, false otherwise.
+ *
+ * The elevel parameter specifies the error level used for logging messages
+ * related to slots that do not exist, are invalidated, or are inactive.
+ */
+bool
+StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel)
+{
+ const char *name;
+ int caught_up_slot_num = 0;
+ XLogRecPtr min_restart_lsn = InvalidXLogRecPtr;
+
+ /*
+ * Don't need to wait for the standbys to catch up if there is no value in
+ * standby_slot_names.
+ */
+ if (standby_slot_names_config == NULL)
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if we are on a standby
+ * server, since we do not support syncing slots to cascading standbys.
+ */
+ if (RecoveryInProgress())
+ return true;
+
+ /*
+ * Don't need to wait for the standbys to catch up if they are already
+ * beyond the specified WAL location.
+ */
+ if (!XLogRecPtrIsInvalid(ss_oldest_flush_lsn) &&
+ ss_oldest_flush_lsn >= wait_for_lsn)
+ return true;
+
+ /*
+ * To prevent concurrent slot dropping and creation while filtering the
+ * slots, take the ReplicationSlotControlLock outside of the loop.
+ */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ name = standby_slot_names_config->slot_names;
+ for (int i = 0; i < standby_slot_names_config->nslotnames; i++)
+ {
+ XLogRecPtr restart_lsn;
+ bool invalidated;
+ bool inactive;
+ ReplicationSlot *slot;
+
+ slot = SearchNamedReplicationSlot(name, false);
+
+ if (!slot)
+ {
+ /*
+ * If a slot name provided in standby_slot_names does not exist,
+ * report a message and exit the loop. A user can specify a slot
+ * name that does not exist just before the server startup. The
+ * GUC check_hook(validate_standby_slots) cannot validate such a
+ * slot during startup as the ReplicationSlotCtl shared memory is
+ * not initialized at that time. It is also possible for a user to
+ * drop the slot in standby_slot_names afterwards.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not exist",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider creating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (SlotIsLogical(slot))
+ {
+ /*
+ * If a logical slot name is provided in standby_slot_names,
+ * report a message and exit the loop. Similar to the non-existent
+ * case, a user can specify a logical slot name in
+ * standby_slot_names before the server startup, or drop an
+ * existing physical slot and recreate a logical slot with the
+ * same name.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot have logical replication slot \"%s\" in parameter %s",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting for correction on \"%s\".",
+ name),
+ errhint("Consider removing logical slot \"%s\" from parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ SpinLockAcquire(&slot->mutex);
+ restart_lsn = slot->data.restart_lsn;
+ invalidated = slot->data.invalidated != RS_INVAL_NONE;
+ inactive = slot->active_pid == 0;
+ SpinLockRelease(&slot->mutex);
+
+ if (invalidated)
+ {
+ /* Specified physical slot has been invalidated */
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("physical slot \"%s\" specified in parameter %s has been invalidated",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider dropping and recreating the slot \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+ break;
+ }
+
+ if (XLogRecPtrIsInvalid(restart_lsn) || restart_lsn < wait_for_lsn)
+ {
+ /* Log a message if no active_pid for this physical slot */
+ if (inactive)
+ ereport(elevel,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("replication slot \"%s\" specified in parameter %s does not have active_pid",
+ name, "standby_slot_names"),
+ errdetail("Logical replication is waiting on the standby associated with \"%s\".",
+ name),
+ errhint("Consider starting standby associated with \"%s\" or amend parameter %s.",
+ name, "standby_slot_names"));
+
+ /* Continue if the current slot hasn't caught up. */
+ break;
+ }
+
+ Assert(restart_lsn >= wait_for_lsn);
+
+ if (XLogRecPtrIsInvalid(min_restart_lsn) ||
+ min_restart_lsn > restart_lsn)
+ min_restart_lsn = restart_lsn;
+
+ caught_up_slot_num++;
+
+ name += strlen(name) + 1;
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /*
+ * Return false if not all the standbys have caught up to the specified
+ * WAL location.
+ */
+ if (caught_up_slot_num != standby_slot_names_config->nslotnames)
+ return false;
+
+ /* The ss_oldest_flush_lsn must not retreat. */
+ Assert(XLogRecPtrIsInvalid(ss_oldest_flush_lsn) ||
+ min_restart_lsn >= ss_oldest_flush_lsn);
+
+ ss_oldest_flush_lsn = min_restart_lsn;
+
+ return true;
+}
+
+/*
+ * Wait for physical standbys to confirm receiving the given lsn.
+ *
+ * Used by logical decoding SQL functions. It waits for physical standbys
+ * corresponding to the physical slots specified in the standby_slot_names GUC.
+ */
+void
+WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
+{
+ /*
+ * Don't need to wait for the standby to catch up if the current acquired
+ * slot is not a logical failover slot, or there is no value in
+ * standby_slot_names.
+ */
+ if (!MyReplicationSlot->data.failover || !standby_slot_names_config)
+ return;
+
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+
+ for (;;)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /* Exit if done waiting for every slot. */
+ if (StandbySlotsHaveCaughtup(wait_for_lsn, WARNING))
+ break;
+
+ /*
+ * Wait for the slots in the standby_slot_names to catch up, but use a
+ * timeout (1s) so we can also check if the standby_slot_names has
+ * been changed.
+ */
+ ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000,
+ WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION);
+ }
+
+ ConditionVariableCancelSleep();
+}
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 768a304723..ad79e1fccd 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -464,6 +464,12 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* crash, but this makes the data consistent after a clean shutdown.
*/
ReplicationSlotMarkDirty();
+
+ /*
+ * Wake up logical walsenders holding logical failover slots after
+ * updating the restart_lsn of the physical slot.
+ */
+ PhysicalWakeupLogicalWalSnd();
}
return retlsn;
@@ -504,6 +510,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto)
.segment_close = wal_segment_close),
NULL, NULL, NULL);
+ /*
+ * Wait for specified streaming replication standby servers (if any)
+ * to confirm receipt of WAL up to moveto lsn.
+ */
+ WaitForStandbyConfirmation(moveto);
+
/*
* Start reading at the slot's restart_lsn, which we know to point to
* a valid record.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 0f1047179c..9d958e6f0b 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1725,26 +1725,111 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
ProcessPendingWrites();
}
+/*
+ * Wake up the logical walsender processes with logical failover slots if the
+ * currently acquired physical slot is specified in standby_slot_names GUC.
+ */
+void
+PhysicalWakeupLogicalWalSnd(void)
+{
+ Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot));
+
+ /*
+ * If we are running in a standby, there is no need to wake up walsenders.
+ * This is because we do not support syncing slots to cascading standbys,
+ * so, there are no walsenders waiting for standbys to catch up.
+ */
+ if (RecoveryInProgress())
+ return;
+
+ if (SlotExistsInStandbySlotNames(NameStr(MyReplicationSlot->data.name)))
+ ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv);
+}
+
+/*
+ * Returns true if not all standbys have caught up to the flushed position
+ * (flushed_lsn) when the current acquired slot is a logical failover
+ * slot and we are streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForStandbys(XLogRecPtr flushed_lsn, uint32 *wait_event)
+{
+ int elevel = got_STOPPING ? ERROR : WARNING;
+ bool failover_slot;
+
+ failover_slot = (replication_active && MyReplicationSlot->data.failover);
+
+ /*
+ * Note that after receiving the shutdown signal, an ERROR is reported if
+ * any slots are dropped, invalidated, or inactive. This measure is taken
+ * to prevent the walsender from waiting indefinitely.
+ */
+ if (failover_slot && !StandbySlotsHaveCaughtup(flushed_lsn, elevel))
+ {
+ *wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION;
+ return true;
+ }
+
+ *wait_event = 0;
+ return false;
+}
+
+/*
+ * Returns true if we need to wait for WALs to be flushed to disk, or if not
+ * all standbys have caught up to the flushed position (flushed_lsn) when the
+ * current acquired slot is a logical failover slot and we are
+ * streaming; otherwise, returns false.
+ *
+ * If returning true, the function sets the appropriate wait event in
+ * wait_event; otherwise, wait_event is set to 0.
+ */
+static bool
+NeedToWaitForWal(XLogRecPtr target_lsn, XLogRecPtr flushed_lsn,
+ uint32 *wait_event)
+{
+ /* Check if we need to wait for WALs to be flushed to disk */
+ if (target_lsn > flushed_lsn)
+ {
+ *wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL;
+ return true;
+ }
+
+ /* Check if the standby slots have caught up to the flushed position */
+ return NeedToWaitForStandbys(flushed_lsn, wait_event);
+}
+
/*
* Wait till WAL < loc is flushed to disk so it can be safely sent to client.
*
- * Returns end LSN of flushed WAL. Normally this will be >= loc, but
- * if we detect a shutdown request (either from postmaster or client)
- * we will return early, so caller must always check.
+ * If the walsender holds a logical slot that has enabled failover, we also
+ * wait for all the specified streaming replication standby servers to confirm
+ * receipt of WAL up to RecentFlushPtr. It is beneficial to wait here for the
+ * confirmation up to RecentFlushPtr rather than waiting before transmitting
+ * each change to logical subscribers, which is already covered by
+ * RecentFlushPtr.
+ *
+ * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we
+ * detect a shutdown request (either from postmaster or client) we will return
+ * early, so caller must always check.
*/
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
int wakeEvents;
+ uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
- * have enough WAL available. This is particularly interesting if we're
- * far behind.
+ * have enough WAL available and all the standby servers have confirmed
+ * receipt of WAL up to RecentFlushPtr. This is particularly interesting
+ * if we're far behind.
*/
- if (RecentFlushPtr != InvalidXLogRecPtr &&
- loc <= RecentFlushPtr)
+ if (!XLogRecPtrIsInvalid(RecentFlushPtr) &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
return RecentFlushPtr;
/* Get a more recent flush pointer. */
@@ -1753,8 +1838,14 @@ WalSndWaitForWal(XLogRecPtr loc)
else
RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * Within the loop, we wait for the necessary WALs to be flushed to disk
+ * first, followed by waiting for standbys to catch up if there are enough
+ * WALs (see NeedToWaitForWal()) or upon receiving the shutdown signal.
+ */
for (;;)
{
+ bool wait_for_standby_at_stop = false;
long sleeptime;
/* Clear any already-pending wakeups */
@@ -1781,21 +1872,35 @@ WalSndWaitForWal(XLogRecPtr loc)
if (got_STOPPING)
XLogBackgroundFlush();
- /* Update our idea of the currently flushed position. */
- if (!RecoveryInProgress())
- RecentFlushPtr = GetFlushRecPtr(NULL);
- else
- RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ /*
+ * To avoid the scenario where standbys need to catch up to a newer
+ * WAL location in each iteration, we update our idea of the currently
+ * flushed position only if we are not waiting for standbys to catch
+ * up.
+ */
+ if (wait_event != WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ {
+ if (!RecoveryInProgress())
+ RecentFlushPtr = GetFlushRecPtr(NULL);
+ else
+ RecentFlushPtr = GetXLogReplayRecPtr(NULL);
+ }
/*
- * If postmaster asked us to stop, don't wait anymore.
+ * If postmaster asked us to stop and the standby slots have caught up
+ * to the flushed position, don't wait anymore.
*
* It's important to do this check after the recomputation of
* RecentFlushPtr, so we can send all remaining data before shutting
* down.
*/
if (got_STOPPING)
- break;
+ {
+ if (NeedToWaitForStandbys(RecentFlushPtr, &wait_event))
+ wait_for_standby_at_stop = true;
+ else
+ break;
+ }
/*
* We only send regular messages to the client for full decoded
@@ -1810,11 +1915,18 @@ WalSndWaitForWal(XLogRecPtr loc)
!waiting_for_ping_response)
WalSndKeepalive(false, InvalidXLogRecPtr);
- /* check whether we're done */
- if (loc <= RecentFlushPtr)
+ /*
+ * Exit the loop if already caught up and doesn't need to wait for
+ * standby slots.
+ */
+ if (!wait_for_standby_at_stop &&
+ !NeedToWaitForWal(loc, RecentFlushPtr, &wait_event))
break;
- /* Waiting for new WAL. Since we need to wait, we're now caught up. */
+ /*
+ * Waiting for new WAL or waiting for standbys to catch up. Since we
+ * need to wait, we're now caught up.
+ */
WalSndCaughtUp = true;
/*
@@ -1852,7 +1964,9 @@ WalSndWaitForWal(XLogRecPtr loc)
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL);
+ Assert(wait_event != 0);
+
+ WalSndWait(wakeEvents, sleeptime, wait_event);
}
/* reactivate latch so WalSndLoop knows to continue */
@@ -2262,6 +2376,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn)
{
ReplicationSlotMarkDirty();
ReplicationSlotsComputeRequiredLSN();
+ PhysicalWakeupLogicalWalSnd();
}
/*
@@ -3535,6 +3650,7 @@ WalSndShmemInit(void)
ConditionVariableInit(&WalSndCtl->wal_flush_cv);
ConditionVariableInit(&WalSndCtl->wal_replay_cv);
+ ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv);
}
}
@@ -3604,8 +3720,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event)
*
* And, we use separate shared memory CVs for physical and logical
* walsenders for selective wake ups, see WalSndWakeup() for more details.
+ *
+ * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV
+ * until awakened by physical walsenders after the walreceiver confirms
+ * the receipt of the LSN.
*/
- if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
+ if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION)
+ ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv);
+ else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv);
else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL)
ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index ec2f31f82a..c08e00d1d6 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP
LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server."
LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server."
SSL_OPEN_SERVER "Waiting for SSL while attempting connection."
+WAIT_FOR_STANDBY_CONFIRMATION "Waiting for WAL to be received and flushed by the physical standby."
WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process."
WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 45013582a7..d77214795d 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4670,6 +4670,20 @@ struct config_string ConfigureNamesString[] =
check_debug_io_direct, assign_debug_io_direct, NULL
},
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("Lists streaming replication standby server slot "
+ "names that logical WAL sender processes will wait for."),
+ gettext_noop("Logical WAL sender processes will send decoded "
+ "changes to plugins only after the specified "
+ "replication slots confirm receiving WAL."),
+ GUC_LIST_INPUT
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, assign_standby_slot_names, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index edcc0282b2..2244ee52f7 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -343,6 +343,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsender processes will wait for
# - Standby Servers -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index acbf567150..425effad21 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -226,6 +226,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
+extern PGDLLIMPORT char *standby_slot_names;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -274,4 +275,8 @@ extern void CheckSlotPermissions(void);
extern ReplicationSlotInvalidationCause
GetSlotInvalidationCause(const char *conflict_reason);
+extern bool SlotExistsInStandbySlotNames(const char *slot_name);
+extern bool StandbySlotsHaveCaughtup(XLogRecPtr wait_for_lsn, int elevel);
+extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn);
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index 0c3996e926..f2d8297f01 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -39,6 +39,7 @@ extern void InitWalSender(void);
extern bool exec_replication_command(const char *cmd_string);
extern void WalSndErrorCleanup(void);
extern void WalSndResourceCleanup(bool isCommit);
+extern void PhysicalWakeupLogicalWalSnd(void);
extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
extern void WalSndSignals(void);
extern Size WalSndShmemSize(void);
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 3113e9ea47..109924ffcd 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -113,6 +113,13 @@ typedef struct
ConditionVariable wal_flush_cv;
ConditionVariable wal_replay_cv;
+ /*
+ * Used by physical walsenders holding slots specified in
+ * standby_slot_names to wake up logical walsenders holding logical
+ * failover slots when a walreceiver confirms the receipt of LSN.
+ */
+ ConditionVariable wal_confirm_rcv_cv;
+
WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER];
} WalSndCtlData;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index c8a7aa9a11..d64dc5fcdb 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra,
extern void assign_wal_consistency_checking(const char *newval, void *extra);
extern bool check_wal_segment_size(int *newval, void **extra, GucSource source);
extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
+extern bool check_standby_slot_names(char **newval, void **extra,
+ GucSource source);
+extern void assign_standby_slot_names(const char *newval, void *extra);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl
index 5c7b4ca5e3..b95d95c06f 100644
--- a/src/test/recovery/t/006_logical_decoding.pl
+++ b/src/test/recovery/t/006_logical_decoding.pl
@@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'},
undef, 'logical slot was actually dropped with DB');
# Test logical slot advancing and its durability.
+# Passing failover=true (last arg) should not have any impact on advancing.
my $logical_slot = 'logical_slot';
$node_primary->safe_psql('postgres',
- "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);"
+ "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);"
);
$node_primary->psql(
'postgres', "
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 021c58f621..99025b352a 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -504,18 +504,258 @@ ok( $standby1->poll_query_until(
"SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot' AND synced AND NOT temporary;"),
'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby');
+##################################################
+# Test that logical failover replication slots wait for the specified
+# physical replication slots to receive the changes first. It uses the
+# following set up:
+#
+# (physical standbys)
+# | ----> standby1 (primary_slot_name = sb1_slot)
+# | ----> standby2 (primary_slot_name = sb2_slot)
+# primary ----- |
+# (logical replication)
+# | ----> subscriber1 (failover = true, slot_name = lsub1_slot)
+# | ----> subscriber2 (failover = false, slot_name = lsub2_slot)
+#
+# standby_slot_names = 'sb1_slot'
+#
+# The setup is configured in such a way that the logical slot of subscriber1 is
+# enabled for failover, and thus the subscriber1 will wait for the physical
+# slot of standby1(sb1_slot) to catch up before receiving the decoded changes.
+##################################################
+
+$backup_name = 'backup3';
+
+$primary->psql('postgres',
+ q{SELECT pg_create_physical_replication_slot('sb2_slot');});
+
+$primary->backup($backup_name);
+
+# Create another standby
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup(
+ $primary, $backup_name,
+ has_streaming => 1,
+ has_restoring => 1);
+$standby2->append_conf(
+ 'postgresql.conf', qq(
+primary_slot_name = 'sb2_slot'
+));
+$standby2->start;
+$primary->wait_for_replay_catchup($standby2);
+
+# Configure primary to disallow any logical slots that have enabled failover
+# from getting ahead of the specified physical replication slot (sb1_slot).
+$primary->append_conf(
+ 'postgresql.conf', qq(
+standby_slot_names = 'sb1_slot'
+));
+$primary->reload;
+
+# Create another subscriber node without enabling failover, wait for sync to
+# complete
+my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$subscriber2->init;
+$subscriber2->start;
+$subscriber2->safe_psql(
+ 'postgres', qq[
+ CREATE TABLE tab_int (a int PRIMARY KEY);
+ CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot);
+]);
+
+$subscriber2->wait_for_subscription_sync;
+
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+my $offset = -s $primary->logfile;
+
+# Stop the standby associated with the specified physical replication slot
+# (sb1_slot) so that the logical replication slot (lsub1_slot) won't receive
+# changes until the standby comes up.
+$standby1->stop;
+
+# Create some data on the primary
+my $primary_row_count = 20;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(11, $primary_row_count);");
+
+# Wait until the standby2 that's still running gets the data from the primary
+$primary->wait_for_replay_catchup($standby2);
+$result = $standby2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby2 gets data from primary");
+
+# Wait for regress_mysub2 to get the data from the primary. This subscription
+# was not enabled for failover so it gets the data without waiting for any
+# standbys.
+$primary->wait_for_catchup('regress_mysub2');
+$result = $subscriber2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "subscriber2 gets data from primary");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# The regress_mysub1 was enabled for failover so it doesn't get the data from
+# primary and keeps waiting for the standby specified in standby_slot_names
+# (sb1_slot aka standby1).
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) <> $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data from primary until standby1 acknowledges changes"
+);
+
+# Start the standby specified in standby_slot_names (sb1_slot aka standby1) and
+# wait for it to catch up with the primary.
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+$result = $standby1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby1 gets data from primary");
+
+# Now that the standby specified in standby_slot_names is up and running, the
+# primary can send the decoded changes to the subscription enabled for failover
+# (i.e. regress_mysub1). While the standby was down, regress_mysub1 didn't
+# receive any data from the primary. i.e. the primary didn't allow it to go
+# ahead of standby.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 acknowledges changes");
+
+##################################################
+# Verify that when using pg_logical_slot_get_changes to consume changes from a
+# logical slot with failover enabled, it will also wait for the slots specified
+# in standby_slot_names to catch up.
+##################################################
+
+# Stop the standby associated with the specified physical replication slot so
+# that the logical replication slot won't receive changes until the standby
+# slot's restart_lsn is advanced or the slot is removed from the
+# standby_slot_names list.
+$primary->safe_psql('postgres', "TRUNCATE tab_int;");
+$primary->wait_for_catchup('regress_mysub1');
+$standby1->stop;
+
+# Disable the regress_mysub1 to prevent the logical walsender from generating
+# more warnings.
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 DISABLE");
+
+# Wait for the replication slot to become inactive on the publisher
+$primary->poll_query_until(
+ 'postgres',
+ "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'",
+ 1);
+
+# Create a logical 'test_decoding' replication slot with failover enabled
+$primary->safe_psql('postgres',
+ "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);"
+);
+
+my $back_q = $primary->background_psql(
+ 'postgres',
+ on_error_stop => 0,
+ timeout => $PostgreSQL::Test::Utils::timeout_default);
+
+# pg_logical_slot_get_changes will be blocked until the standby catches up,
+# hence it needs to be executed in a background session.
+$offset = -s $primary->logfile;
+$back_q->query_until(
+ qr/logical_slot_get_changes/, q(
+ \echo logical_slot_get_changes
+ SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);
+));
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the function
+# pg_logical_slot_get_changes should now return, and the session can be
+# stopped.
+$back_q->quit;
+
+$primary->safe_psql('postgres',
+ "SELECT pg_drop_replication_slot('test_slot');"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
+# Enable the regress_mysub1 for further tests
+$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE");
+
+##################################################
+# Test that logical replication will wait for the user-created inactive
+# physical slot to catch up until we remove the slot from standby_slot_names.
+##################################################
+
+$offset = -s $primary->logfile;
+
+# Create some data on the primary
+$primary_row_count = 10;
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);");
+
+# Wait until the primary server logs a warning indicating that it is waiting
+# for the sb1_slot to catch up.
+$primary->wait_for_log(
+ qr/replication slot \"sb1_slot\" specified in parameter standby_slot_names does not have active_pid/,
+ $offset);
+
+# The regress_mysub1 doesn't get the data from primary because the specified
+# standby slot (sb1_slot) in standby_slot_names is inactive.
+$result =
+ $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;");
+is($result, 't',
+ "subscriber1 doesn't get data as the sb1_slot doesn't catch up");
+
+# Remove the standby from the standby_slot_names list and reload the
+# configuration.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''");
+$primary->reload;
+
+# Since there are no slots in standby_slot_names, the primary server should now
+# send the decoded changes to the subscription.
+$primary->wait_for_catchup('regress_mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't',
+ "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list"
+);
+
+# Add the physical slot (sb1_slot) back to the standby_slot_names for further
+# tests.
+$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "'sb1_slot'");
+$primary->reload;
+
##################################################
# Promote the standby1 to primary. Confirm that:
# a) the slot 'lsub1_slot' is retained on the new primary
# b) logical replication for regress_mysub1 is resumed successfully after failover
##################################################
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+
$standby1->promote;
# Update subscription with the new primary's connection info
my $standby1_conninfo = $standby1->connstr . ' dbname=postgres';
$subscriber1->safe_psql('postgres',
- "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo';
- ALTER SUBSCRIPTION regress_mysub1 ENABLE; ");
+ "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo';");
# Confirm the synced slot 'lsub1_slot' is retained on the new primary
is($standby1->safe_psql('postgres',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 95ae7845d8..57fb6a0c84 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2650,6 +2650,7 @@ SplitPoint
SplitTextOutputData
SplitVar
StackElem
+StandbySlotNamesConfigData
StartDataPtrType
StartLOPtrType
StartLOsPtrType
--
2.30.0.windows.2
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-08 03:33 Amit Kapila <[email protected]>
parent: Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Amit Kapila @ 2024-03-08 03:33 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Peter Smith <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Thu, Mar 7, 2024 at 12:00 PM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
>
> Attach the V108 patch set which addressed above and Peter's comments.
> I also removed the check for "*" in guc check hook.
>
Pushed with minor modifications. I'll keep an eye on BF.
BTW, one thing that we should try to evaluate a bit more is the
traversal of slots in StandbySlotsHaveCaughtup() where we verify if
all the slots mentioned in standby_slot_names have received the
required WAL. Even if the standby_slot_names list is short the total
number of slots can be much larger which can lead to an increase in
CPU usage during traversal. There is an optimization that allows to
cache ss_oldest_flush_lsn and ensures that we don't need to traverse
the slots each time so it may not hit frequently but still there is a
chance. I see it is possible to further optimize this area by caching
the position of each slot mentioned in standby_slot_names in
replication_slots array but not sure whether it is worth.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-08 04:26 Ajin Cherian <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 68+ messages in thread
From: Ajin Cherian @ 2024-03-08 04:26 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>
On Fri, Mar 8, 2024 at 2:33 PM Amit Kapila <[email protected]> wrote:
> On Thu, Mar 7, 2024 at 12:00 PM Zhijie Hou (Fujitsu)
> <[email protected]> wrote:
> >
> >
> > Attach the V108 patch set which addressed above and Peter's comments.
> > I also removed the check for "*" in guc check hook.
> >
>
>
> Pushed with minor modifications. I'll keep an eye on BF.
>
> BTW, one thing that we should try to evaluate a bit more is the
> traversal of slots in StandbySlotsHaveCaughtup() where we verify if
> all the slots mentioned in standby_slot_names have received the
> required WAL. Even if the standby_slot_names list is short the total
> number of slots can be much larger which can lead to an increase in
> CPU usage during traversal. There is an optimization that allows to
> cache ss_oldest_flush_lsn and ensures that we don't need to traverse
> the slots each time so it may not hit frequently but still there is a
> chance. I see it is possible to further optimize this area by caching
> the position of each slot mentioned in standby_slot_names in
> replication_slots array but not sure whether it is worth.
>
>
>
I tried to test this by configuring a large number of logical slots while
making sure the standby slots are at the end of the array and checking if
there was any performance hit in logical replication from these searches.
Setup:
1. 1 primary server configured with 3 servers in the standby_slot_names, 1
extra logical slot (not configured for failover) + 1 logical subscriber
configures as failover + 3 physical standbys(all configured to sync logical
slots)
2. 1 primary server configured with 3 servers in the standby_slot_names,
100 extra logical slot (not configured for failover) + 1 logical subscriber
configures as failover + 3 physical standbys(all configured to sync logical
slots)
3. 1 primary server configured with 3 servers in the standby_slot_names,
500 extra logical slot (not configured for failover) + 1 logical subscriber
configures as failover + 3 physical standbys(all configured to sync logical
slots)
In the three setups, 3 standby_slot_names are compared with a list of 2,101
and 501 slots respectively.
I ran a pgbench for 15 minutes for all 3 setups:
Case 1: Average TPS - 8.143399 TPS
Case 2: Average TPS - 8.187462 TPS
Case 3: Average TPS - 8.190611 TPS
I see no degradation in the performance, the differences in performance are
well within the run to run variations seen.
Nisha also did some performance tests to record the lag introduced by the
large number of slots traversal in StandbySlotsHaveCaughtup(). The tests
logged time at the start and end of the XLogSendLogical() call (which
eventually calls WalSndWaitForWal() --> StandbySlotsHaveCaughtup()) and
calculated total time taken by this function during the load run for
different total slots count.
Setup:
--one primary with 3 standbys and one subscriber with one active
subscription
--hot_standby_feedback=off and sync_replication_slots=false
--made sure the standby slots remain at the end
ReplicationSlotCtl->replication_slots array to measure performance of worst
case scenario for standby slot search in StandbySlotsHaveCaughtup()
pgbench for 15 min was run. Here is the data:
Case1 : with 1 logical slot, standby_slot_names having 3 slots
Run1: 626.141642 secs
Run2: 631.930254 secs
Case2 : with 100 logical slots, standby_slot_names having 3 slots
Run1: 629.38332 secs
Run2: 630.548432 secs
Case3 : with 500 logical slots, standby_slot_names having 3 slots
Run1: 629.910829 secs
Run2: 627.924183 secs
There was no degradation in performance seen.
Thanks Nisha for helping with the testing.
regards,
Ajin Cherian
Fujitsu Australia
^ permalink raw reply [nested|flat] 68+ messages in thread
* Re: Synchronizing slots from primary to standby
@ 2024-03-08 05:09 shveta malik <[email protected]>
parent: Ajin Cherian <[email protected]>
0 siblings, 0 replies; 68+ messages in thread
From: shveta malik @ 2024-03-08 05:09 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: Amit Kapila <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Masahiko Sawada <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bertrand Drouvot <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>; shveta malik <[email protected]>
On Fri, Mar 8, 2024 at 9:56 AM Ajin Cherian <[email protected]> wrote:
>
>> Pushed with minor modifications. I'll keep an eye on BF.
>>
>> BTW, one thing that we should try to evaluate a bit more is the
>> traversal of slots in StandbySlotsHaveCaughtup() where we verify if
>> all the slots mentioned in standby_slot_names have received the
>> required WAL. Even if the standby_slot_names list is short the total
>> number of slots can be much larger which can lead to an increase in
>> CPU usage during traversal. There is an optimization that allows to
>> cache ss_oldest_flush_lsn and ensures that we don't need to traverse
>> the slots each time so it may not hit frequently but still there is a
>> chance. I see it is possible to further optimize this area by caching
>> the position of each slot mentioned in standby_slot_names in
>> replication_slots array but not sure whether it is worth.
>>
>>
>
> I tried to test this by configuring a large number of logical slots while making sure the standby slots are at the end of the array and checking if there was any performance hit in logical replication from these searches.
>
Thanks Ajin and Nisha.
We also plan:
1) Redoing XLogSendLogical time-log related test with
'sync_replication_slots' enabled.
2) pg_recvlogical test to monitor lag in StandbySlotsHaveCaughtup()
for a large number of slots.
3) Profiling to see if StandbySlotsHaveCaughtup() is noticeable in the
report when there are a large number of slots to traverse.
thanks
Shveta
^ permalink raw reply [nested|flat] 68+ messages in thread
end of thread, other threads:[~2024-03-08 05:09 UTC | newest]
Thread overview: 68+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-01-11 06:00 [PATCH 5/6] Documentation for slot-limit feature Kyotaro Horiguchi <[email protected]>
2019-12-20 01:21 [PATCH v29 05/11] Add Incremental View Maintenance support to psql Yugo Nagata <[email protected]>
2024-02-27 01:42 RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-02-27 07:17 ` Re: Synchronizing slots from primary to standby Peter Smith <[email protected]>
2024-02-27 10:37 ` Re: Synchronizing slots from primary to standby Bertrand Drouvot <[email protected]>
2024-02-27 11:59 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-02-27 12:35 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-02-29 03:43 ` Re: Synchronizing slots from primary to standby Peter Smith <[email protected]>
2024-02-29 10:08 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-03-04 08:57 ` Re: Synchronizing slots from primary to standby Bertrand Drouvot <[email protected]>
2024-03-05 03:24 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-02-28 02:23 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-02-28 11:35 ` Re: Synchronizing slots from primary to standby Bertrand Drouvot <[email protected]>
2024-02-29 03:16 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-02 03:51 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-02-29 02:59 ` Re: Synchronizing slots from primary to standby Peter Smith <[email protected]>
2024-02-29 09:53 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-02-29 11:17 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-03-01 02:17 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-01 03:42 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-01 06:10 ` Re: Synchronizing slots from primary to standby Masahiko Sawada <[email protected]>
2024-03-01 06:21 ` Re: Synchronizing slots from primary to standby Peter Smith <[email protected]>
2024-03-06 06:37 ` Re: Synchronizing slots from primary to standby Masahiko Sawada <[email protected]>
2024-03-06 08:53 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-03-07 03:19 ` Re: Synchronizing slots from primary to standby Masahiko Sawada <[email protected]>
2024-03-01 07:21 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-06 01:30 ` Re: Synchronizing slots from primary to standby Masahiko Sawada <[email protected]>
2024-03-06 03:03 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-06 13:13 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-06 13:24 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-07 02:04 ` Re: Synchronizing slots from primary to standby Peter Smith <[email protected]>
2024-03-07 04:46 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-03-07 06:30 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-08 03:33 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-03-08 04:26 ` Re: Synchronizing slots from primary to standby Ajin Cherian <[email protected]>
2024-03-08 05:09 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2024-03-07 06:28 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-07 03:07 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2024-03-07 05:54 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-03-01 09:21 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-03-01 04:22 ` Re: Synchronizing slots from primary to standby Peter Smith <[email protected]>
2024-03-01 06:45 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-03-01 07:15 ` Re: Synchronizing slots from primary to standby Bertrand Drouvot <[email protected]>
2024-03-02 03:51 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-02 10:54 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-03-03 07:56 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-04 11:22 ` Re: Synchronizing slots from primary to standby Bertrand Drouvot <[email protected]>
2024-03-04 13:26 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-02 23:46 ` Re: Synchronizing slots from primary to standby Peter Smith <[email protected]>
2024-03-03 03:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-03-04 01:27 ` Re: Synchronizing slots from primary to standby Peter Smith <[email protected]>
2024-03-04 03:38 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-03-04 04:05 ` Re: Synchronizing slots from primary to standby Peter Smith <[email protected]>
2024-03-04 09:51 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-03-04 13:28 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-04 15:43 ` Re: Synchronizing slots from primary to standby Bertrand Drouvot <[email protected]>
2024-03-05 07:15 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-05 00:40 ` Re: Synchronizing slots from primary to standby Peter Smith <[email protected]>
2024-03-05 03:45 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-03-05 06:35 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2024-03-05 06:45 ` Re: Synchronizing slots from primary to standby Nisha Moond <[email protected]>
2024-03-05 07:21 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-03 07:51 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-04 01:55 ` Re: Synchronizing slots from primary to standby Peter Smith <[email protected]>
2024-03-04 03:49 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2024-03-04 03:57 ` Re: Synchronizing slots from primary to standby Peter Smith <[email protected]>
2024-03-04 13:26 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
2024-03-01 02:18 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox