public inbox for [email protected]
help / color / mirror / Atom feed[PATCH] pgbench: increase the maximum number of variables/arguments
74+ messages / 9 participants
[nested] [flat]
* [PATCH] pgbench: increase the maximum number of variables/arguments
@ 2019-03-10 23:20 Dagfinn Ilmari Mannsåker <[email protected]>
0 siblings, 0 replies; 74+ messages in thread
From: Dagfinn Ilmari Mannsåker @ 2019-03-10 23:20 UTC (permalink / raw)
pgbench has a strange restriction to only allow 10 arguments, which is
too low for some real world uses.
Since MAX_ARGS is only used for an array of pointers and an array of
integers increasing this should not be a problem, so increase value to 256.
Add coments to clarify that MAX_ARGS includes the SQL statement or
backslash metacommand, respectively, since this caused some confusion as
to whether there was an off-by-one error in the error checking and
message.
---
doc/src/sgml/ref/pgbench.sgml | 2 ++
src/bin/pgbench/pgbench.c | 15 ++++++++++-
src/bin/pgbench/t/001_pgbench_with_server.pl | 28 ++++----------------
3 files changed, 21 insertions(+), 24 deletions(-)
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 9d18524834..e1ab73e582 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -916,6 +916,8 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
value can be inserted into a SQL command by writing
<literal>:</literal><replaceable>variablename</replaceable>. When running more than
one client session, each session has its own set of variables.
+ <application>pgbench</application> supports up to 255 variables in one
+ statement.
</para>
<table id="pgbench-automatic-variables">
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 5df54a8e57..4789ab92ee 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -476,7 +476,12 @@ typedef struct
*/
#define SQL_COMMAND 1
#define META_COMMAND 2
-#define MAX_ARGS 10
+
+/*
+ * max number of backslash command arguments or SQL variables,
+ * including the command or SQL statement itself
+ */
+#define MAX_ARGS 256
typedef enum MetaCommand
{
@@ -4124,6 +4129,10 @@ parseQuery(Command *cmd)
continue;
}
+ /*
+ * cmd->argv[0] is the SQL statement itself, so the max number of
+ * arguments is one less than MAX_ARGS
+ */
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
@@ -4461,6 +4470,10 @@ process_backslash_command(PsqlScanState sstate, const char *source)
/* For all other commands, collect remaining words. */
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
+ /*
+ * my_command->argv[0] is the command itself, so the max number of
+ * arguments is one less than MAX_ARGS
+ */
if (j >= MAX_ARGS)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 930ff4ebb9..6b586210a2 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -597,11 +597,10 @@ my @errors = (
}
],
[
- 'sql too many args', 1, [qr{statement has too many arguments.*\b9\b}],
- q{-- MAX_ARGS=10 for prepared
+ 'sql too many args', 1, [qr{statement has too many arguments.*\b255\b}],
+ q{-- MAX_ARGS=256 for prepared
\set i 0
-SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
-}
+SELECT LEAST(}.join(', ', (':i') x 256).q{)}
],
# SHELL
@@ -619,25 +618,8 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[ 'shell missing command', 1, [qr{missing command }], q{\shell} ],
[
'shell too many args', 1, [qr{too many arguments in command "shell"}],
- q{-- 257 arguments to \shell
-\shell echo \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F \
- 0 1 2 3 4 5 6 7 8 9 A B C D E F
-}
+ q{-- 256 arguments to \shell
+\shell }.join(' ', ('echo') x 256)
],
# SET
--
2.21.0
--=-=-=--
^ permalink raw reply [nested|flat] 74+ messages in thread
* Optionally automatically disable logical replication subscriptions on error
@ 2021-06-17 20:18 Mark Dilger <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: Mark Dilger @ 2021-06-17 20:18 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>
Hackers,
Logical replication apply workers for a subscription can easily get stuck in an infinite loop of attempting to apply a change, triggering an error (such as a constraint violation), exiting with an error written to the subscription worker log, and restarting.
As things currently stand, only superusers can create subscriptions. Ongoing work to delegate superuser tasks to non-superusers creates the potential for even more errors to be triggered, specifically, errors where the apply worker does not have permission to make changes to the target table.
The attached patch makes it possible to create a subscription using a new subscription_parameter, "disable_on_error", such that rather than going into an infinite loop, the apply worker will catch errors and automatically disable the subscription, breaking the loop. The new parameter defaults to false. When false, the PG_TRY/PG_CATCH overhead is avoided, so for subscriptions which do not use the feature, there shouldn't be any change. Users can manually clear the error after fixing the underlying issue with an ALTER SUBSCRIPTION .. ENABLE command.
In addition to helping on production systems, this makes writing TAP tests involving error conditions simpler. I originally ran into the motivation to write this patch when frustrated that TAP tests needed to parse the apply worker log file to determine whether permission failures were occurring and what they were. It was also obnoxiously easy to have a test get stuck waiting for a permanently stuck subscription to catch up. This helps with both issues.
I don't think this is quite ready for commit, but I'd like feedback if folks like this idea or want to suggest design changes.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
[application/octet-stream] v1-0001-Optionally-disabling-subscriptions-on-error.patch (27.1K, ../../[email protected]/2-v1-0001-Optionally-disabling-subscriptions-on-error.patch)
download | inline diff:
From e046be4b851ccebddabae47f57cbc0d8f49fadc8 Mon Sep 17 00:00:00 2001
From: Mark Dilger <[email protected]>
Date: Thu, 17 Jun 2021 12:39:19 -0700
Subject: [PATCH v1] Optionally disabling subscriptions on error
Logical replication apply workers for a subscription can easily get
stuck in an infinite loop of attempting to apply a change,
triggering an error (such as a constraint violation), exiting with
an error written to the subscription worker log, and restarting.
To partially remedy the situation, adding a new
subscription_parameter named 'disable_on_error'. To be consistent
with old behavior, the parameter defaults to false. When true,
the apply worker catches errors thrown, and for errors that are
deemed not to be transient, disables the subscription in order to
break the loop. New columns in the pg_subscription table help
diagnose the situation: 'disabled_by_error' shows whether this has
occurred and 'suberrmsg' includes the messsage field of the error.
The error is still also written to the logs.
In addition to helping on production systems, this makes writing TAP
tests involving error conditions simpler. Rather than having to
open and parse the apply worker's log file, the test can query the
pg_subscription table. It also helps that the workers don't go into
an infinite loop during the test.
---
doc/src/sgml/catalogs.sgml | 10 +
src/backend/catalog/pg_subscription.c | 10 +
src/backend/catalog/system_views.sql | 8 +-
src/backend/commands/subscriptioncmds.c | 62 ++++++
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/worker.c | 176 +++++++++++++++++-
src/bin/pg_dump/pg_dump.c | 6 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/include/catalog/pg_subscription.h | 13 ++
.../subscription/t/022_disable_on_error.pl | 125 +++++++++++++
10 files changed, 408 insertions(+), 5 deletions(-)
create mode 100644 src/test/subscription/t/022_disable_on_error.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index f517a7d4af..6fd95cb7ca 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7671,6 +7671,16 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>suberrmsg</structfield> <type>text</type>
+ </para>
+ <para>
+ The message from the error which disabled the subscription, if it has
+ been automatically disabled.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subpublications</structfield> <type>text[]</type>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 29fc4218cd..20b51a5f1f 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,8 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disable_on_error = subform->disable_on_error;
+ sub->disabled_by_error = subform->disabled_by_error;
sub->binary = subform->subbinary;
sub->stream = subform->substream;
@@ -95,6 +97,14 @@ GetSubscription(Oid subid, bool missing_ok)
Assert(!isnull);
sub->synccommit = TextDatumGetCString(datum);
+ /* Get errmsg */
+ datum = SysCacheGetAttr(SUBSCRIPTIONOID,
+ tup,
+ Anum_pg_subscription_suberrmsg,
+ &isnull);
+ Assert(!isnull);
+ sub->errmsg = TextDatumGetCString(datum);
+
/* Get publications */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
tup,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 999d984068..49a02f9353 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1252,8 +1252,10 @@ CREATE VIEW pg_replication_origin_status AS
REVOKE ALL ON pg_replication_origin_status FROM public;
--- All columns of pg_subscription except subconninfo are publicly readable.
+-- All columns of pg_subscription except subconninfo and suberrmsg are publicly
+-- readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
- substream, subslotname, subsynccommit, subpublications)
+GRANT SELECT (oid, subdbid, subname, subowner, subenabled, disable_on_error,
+ disabled_by_error, subbinary, substream, subslotname,
+ subsynccommit, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 75e195f286..098036f77e 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -63,6 +63,8 @@ static void
parse_subscription_options(List *options,
bool *connect,
bool *enabled_given, bool *enabled,
+ bool *disable_on_error_given,
+ bool *disable_on_error,
bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data,
@@ -84,13 +86,24 @@ parse_subscription_options(List *options,
*connect = true;
if (enabled)
{
+ Assert(enabled_given);
+
*enabled_given = false;
*enabled = true;
}
+ if (disable_on_error)
+ {
+ Assert(disable_on_error_given);
+
+ *disable_on_error_given = false;
+ *disable_on_error = false;
+ }
if (create_slot)
*create_slot = true;
if (slot_name)
{
+ Assert(slot_name_given);
+
*slot_name_given = false;
*slot_name = NULL;
}
@@ -102,11 +115,15 @@ parse_subscription_options(List *options,
*refresh = true;
if (binary)
{
+ Assert(binary_given);
+
*binary_given = false;
*binary = false;
}
if (streaming)
{
+ Assert(streaming_given);
+
*streaming_given = false;
*streaming = false;
}
@@ -136,6 +153,16 @@ parse_subscription_options(List *options,
*enabled_given = true;
*enabled = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "disable_on_error") == 0 &&
+ disable_on_error)
+ {
+ if (*disable_on_error_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ *disable_on_error_given = true;
+ *disable_on_error = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "create_slot") == 0 && create_slot)
{
if (create_slot_given)
@@ -334,6 +361,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool connect;
bool enabled_given;
bool enabled;
+ bool disable_on_error_given;
+ bool disable_on_error;
bool copy_data;
bool streaming;
bool streaming_given;
@@ -355,6 +384,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options,
&connect,
&enabled_given, &enabled,
+ &disable_on_error_given,
+ &disable_on_error,
&create_slot,
&slotname_given, &slotname,
©_data,
@@ -427,6 +458,10 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_disable_on_error - 1] =
+ BoolGetDatum(disable_on_error);
+ values[Anum_pg_subscription_disabled_by_error - 1] =
+ BoolGetDatum(false);
values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_substream - 1] = BoolGetDatum(streaming);
values[Anum_pg_subscription_subconninfo - 1] =
@@ -438,6 +473,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
nulls[Anum_pg_subscription_subslotname - 1] = true;
values[Anum_pg_subscription_subsynccommit - 1] =
CStringGetTextDatum(synchronous_commit);
+ values[Anum_pg_subscription_suberrmsg - 1] =
+ CStringGetTextDatum("");
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
@@ -799,6 +836,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
{
case ALTER_SUBSCRIPTION_OPTIONS:
{
+ bool disable_on_error;
+ bool disable_on_error_given;
char *slotname;
bool slotname_given;
char *synchronous_commit;
@@ -810,6 +849,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options,
NULL, /* no "connect" */
NULL, NULL, /* no "enabled" */
+ &disable_on_error_given,
+ &disable_on_error,
NULL, /* no "create_slot" */
&slotname_given, &slotname,
NULL, /* no "copy_data" */
@@ -818,6 +859,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
&binary_given, &binary,
&streaming_given, &streaming);
+ if (disable_on_error_given)
+ {
+ values[Anum_pg_subscription_disable_on_error - 1] =
+ BoolGetDatum(disable_on_error);
+ replaces[Anum_pg_subscription_disable_on_error - 1 ] =
+ true;
+ }
+
if (slotname_given)
{
if (sub->enabled && !slotname)
@@ -867,6 +916,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options,
NULL, /* no "connect" */
&enabled_given, &enabled,
+ NULL, NULL, /* no "disable_on_error" */
NULL, /* no "create_slot" */
NULL, NULL, /* no "slot_name" */
NULL, /* no "copy_data" */
@@ -885,6 +935,15 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
BoolGetDatum(enabled);
replaces[Anum_pg_subscription_subenabled - 1] = true;
+ /*
+ * Manually enabling or disabling a subscription clears any
+ * error which automatically disabled it.
+ */
+ values[Anum_pg_subscription_disabled_by_error - 1] = false;
+ replaces[Anum_pg_subscription_disabled_by_error - 1] = true;
+ values[Anum_pg_subscription_suberrmsg - 1] = CStringGetTextDatum("");
+ replaces[Anum_pg_subscription_suberrmsg - 1] = true;
+
if (enabled)
ApplyLauncherWakeupAtCommit();
@@ -912,6 +971,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options,
NULL, /* no "connect" */
NULL, NULL, /* no "enabled" */
+ NULL, NULL, /* no "disable_on_error" */
NULL, /* no "create_slot" */
NULL, NULL, /* no "slot_name" */
©_data,
@@ -958,6 +1018,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options,
NULL, /* no "connect" */
NULL, NULL, /* no "enabled" */
+ NULL, NULL, /* no "disable_on_error" */
NULL, /* no "create_slot" */
NULL, NULL, /* no "slot_name" */
isadd ? ©_data : NULL, /* for drop, no
@@ -1005,6 +1066,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options,
NULL, /* no "connect" */
NULL, NULL, /* no "enabled" */
+ NULL, NULL, /* no "disable_on_error" */
NULL, /* no "create_slot" */
NULL, NULL, /* no "slot_name" */
©_data,
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index e3b11daa89..e8a32b13b6 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -132,6 +132,8 @@ get_subscription_list(void)
sub->dbid = subform->subdbid;
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disable_on_error = subform->disable_on_error;
+ sub->disabled_by_error = subform->disabled_by_error;
sub->name = pstrdup(NameStr(subform->subname));
/* We don't fill fields we are not interested in. */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index bbb659dad0..ffd57b7efc 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -62,6 +62,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/partition.h"
#include "catalog/pg_inherits.h"
@@ -2412,6 +2413,162 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
walrcv_endstreaming(LogRepWorkerWalRcvConn, &tli);
}
+/*
+ * Apply main loop with logic to disable a stuck subscription
+ *
+ * Applies changes, and for non-transient errors, catches the error and
+ * disables the subscription before rethrowing.
+ */
+static void
+LogicalRepApplyLoopTry(XLogRecPtr last_received)
+{
+ MemoryContext mcxt = CurrentMemoryContext;
+ bool did_error = false;
+
+ PG_TRY();
+ {
+ LogicalRepApplyLoop(last_received);
+ }
+ PG_CATCH();
+ {
+ /*
+ * If the error is transient, network protocol related, or resource
+ * exhaustion related, just rethrow it. We only want to disable the
+ * subscription for permanent errors.
+ */
+ switch (geterrcode())
+ {
+ case ERRCODE_CONNECTION_EXCEPTION:
+ case ERRCODE_CONNECTION_DOES_NOT_EXIST:
+ case ERRCODE_CONNECTION_FAILURE:
+ case ERRCODE_SQLCLIENT_UNABLE_TO_ESTABLISH_SQLCONNECTION:
+ case ERRCODE_SQLSERVER_REJECTED_ESTABLISHMENT_OF_SQLCONNECTION:
+ case ERRCODE_TRANSACTION_RESOLUTION_UNKNOWN:
+ case ERRCODE_PROTOCOL_VIOLATION:
+ case ERRCODE_INSUFFICIENT_RESOURCES:
+ case ERRCODE_DISK_FULL:
+ case ERRCODE_OUT_OF_MEMORY:
+ case ERRCODE_TOO_MANY_CONNECTIONS:
+ case ERRCODE_CONFIGURATION_LIMIT_EXCEEDED:
+ case ERRCODE_PROGRAM_LIMIT_EXCEEDED:
+ case ERRCODE_STATEMENT_TOO_COMPLEX:
+ case ERRCODE_TOO_MANY_COLUMNS:
+ case ERRCODE_TOO_MANY_ARGUMENTS:
+ case ERRCODE_OPERATOR_INTERVENTION:
+ case ERRCODE_QUERY_CANCELED:
+ case ERRCODE_ADMIN_SHUTDOWN:
+ case ERRCODE_CRASH_SHUTDOWN:
+ case ERRCODE_CANNOT_CONNECT_NOW:
+ case ERRCODE_DATABASE_DROPPED:
+ case ERRCODE_IDLE_SESSION_TIMEOUT:
+ PG_RE_THROW();
+ default:
+ break;
+ }
+
+ /*
+ * Record that we had an error, but otherwise don't do anything here in the
+ * catch block, as anything complicated we do might itself throw.
+ */
+ did_error = true;
+ }
+ PG_END_TRY();
+
+ /*
+ * If we caught an error above, disable the subscription and record copies
+ * of the relevant error text.
+ */
+ if (did_error)
+ {
+ Relation rel;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Oid subid;
+ Form_pg_subscription form;
+ ErrorData *edata;
+
+ /*
+ * Clean up from the error and get a fresh transaction in which to
+ * disable the subscription.
+ */
+ MemoryContextSwitchTo(mcxt);
+ edata = CopyErrorData();
+
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+
+ StartTransactionCommand();
+
+ /* Look up our subscription in the catalogs */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy2(SUBSCRIPTIONNAME, MyDatabaseId,
+ CStringGetDatum(MySubscription->name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("subscription \"%s\" does not exist",
+ MySubscription->name)));
+
+ form = (Form_pg_subscription) GETSTRUCT(tup);
+ subid = form->oid;
+ LockSharedObject(SubscriptionRelationId, subid, 0, AccessExclusiveLock);
+
+ /*
+ * We would not be here unless this subscription's disable_on_error
+ * field was true when our worker began applying changes, but check
+ * whether that field has changed in the interim.
+ */
+ if (!form->disable_on_error)
+ ReThrowError(edata);
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Set the subscription to disabled, and note the reason. */
+ values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(false);
+ replaces[Anum_pg_subscription_subenabled - 1] = true;
+ values[Anum_pg_subscription_disabled_by_error - 1] =
+ BoolGetDatum(true);
+ replaces[Anum_pg_subscription_disabled_by_error - 1] = true;
+ values[Anum_pg_subscription_suberrmsg - 1] =
+ CStringGetTextDatum(edata->message ? edata->message : "");
+ replaces[Anum_pg_subscription_suberrmsg - 1] = true;
+
+ /* Update the catalog */
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+ heap_freetuple(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ ereport(LOG,
+ (errmsg("edata is true for subscription '%s': message = \"%s\", hint = \"%s\"",
+ MySubscription->name,
+ edata->message ? edata->message : "<NONE>",
+ edata->hint ? edata->hint : "<NONE>")));
+
+ ereport(LOG,
+ (errmsg("logical replication apply worker for subscription \"%s\" will "
+ "stop because the subscription was disabled due to error",
+ MySubscription->name)));
+
+ CommitTransactionCommand();
+
+ /*
+ * Now that we've successfully disabled the subscription, rethrow the
+ * error for any other consumers who might want to see it. In
+ * particular, we expect that the worker will write the error to the
+ * server log and then exit.
+ */
+ ReThrowError(edata);
+ }
+}
+
/*
* Send a Standby Status Update message to server.
*
@@ -3138,6 +3295,20 @@ ApplyWorkerMain(Datum main_arg)
proc_exit(0);
}
+ /*
+ * This is probably unnecessary, as an error which sets "disabled_by_error" will also
+ * have cleared "enabled", but for now, see if this ever resolves to true.
+ */
+ if (MySubscription->disabled_by_error)
+ {
+ ereport(LOG,
+ (errmsg("logical replication apply worker for subscription \"%s\" will not "
+ "start because the subscription was disabled by an error",
+ MySubscription->name)));
+
+ proc_exit(0);
+ }
+
/* Setup synchronous commit according to the user's wishes */
SetConfigOption("synchronous_commit", MySubscription->synccommit,
PGC_BACKEND, PGC_S_OVERRIDE);
@@ -3241,7 +3412,10 @@ ApplyWorkerMain(Datum main_arg)
walrcv_startstreaming(LogRepWorkerWalRcvConn, &options);
/* Run the main loop. */
- LogicalRepApplyLoop(origin_startpos);
+ if (MySubscription->disable_on_error)
+ LogicalRepApplyLoopTry(origin_startpos);
+ else
+ LogicalRepApplyLoop(origin_startpos);
proc_exit(0);
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 8f53cc7c3b..16d9d006d1 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4307,6 +4307,7 @@ getSubscriptions(Archive *fout)
int i_subconninfo;
int i_subslotname;
int i_subsynccommit;
+ int i_suberrmsg;
int i_subpublications;
int i_subbinary;
int i,
@@ -4338,7 +4339,7 @@ getSubscriptions(Archive *fout)
"SELECT s.tableoid, s.oid, s.subname,\n"
" (%s s.subowner) AS rolname,\n"
" s.subconninfo, s.subslotname, s.subsynccommit,\n"
- " s.subpublications,\n",
+ " s.suberrmsg, s.subpublications,\n",
username_subquery);
if (fout->remoteVersion >= 140000)
@@ -4367,6 +4368,7 @@ getSubscriptions(Archive *fout)
i_subconninfo = PQfnumber(res, "subconninfo");
i_subslotname = PQfnumber(res, "subslotname");
i_subsynccommit = PQfnumber(res, "subsynccommit");
+ i_suberrmsg = PQfnumber(res, "suberrmsg");
i_subpublications = PQfnumber(res, "subpublications");
i_subbinary = PQfnumber(res, "subbinary");
i_substream = PQfnumber(res, "substream");
@@ -4389,6 +4391,8 @@ getSubscriptions(Archive *fout)
subinfo[i].subslotname = pg_strdup(PQgetvalue(res, i, i_subslotname));
subinfo[i].subsynccommit =
pg_strdup(PQgetvalue(res, i, i_subsynccommit));
+ subinfo[i].suberrmsg =
+ pg_strdup(PQgetvalue(res, i, i_suberrmsg));
subinfo[i].subpublications =
pg_strdup(PQgetvalue(res, i, i_subpublications));
subinfo[i].subbinary =
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 49e1b0a09c..3c2a8dd38c 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -638,6 +638,7 @@ typedef struct _SubscriptionInfo
char *subbinary;
char *substream;
char *subsynccommit;
+ char *suberrmsg;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0060ebfb40..7ffa41c26b 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -52,6 +52,12 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool disable_on_error; /* True if apply errors should automatically
+ disable the subscription */
+
+ bool disabled_by_error; /* True if an error caused the subscription
+ to be disabled */
+
bool subbinary; /* True if the subscription wants the
* publisher to send data in binary */
@@ -67,6 +73,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
/* Synchronous commit setting for worker */
text subsynccommit BKI_FORCE_NOT_NULL;
+ /* Message from error which disabled this subscription */
+ text suberrmsg BKI_FORCE_NOT_NULL;
+
/* List of publications subscribed to */
text subpublications[1] BKI_FORCE_NOT_NULL;
#endif
@@ -91,12 +100,16 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool disable_on_error; /* Whether errors automatically disable */
+ bool disabled_by_error; /* Whether an error has disabled */
bool binary; /* Indicates if the subscription wants data in
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
+ char *errmsg; /* Message from error which disabled */
+ char *errhint; /* Hint from error which disabled */
List *publications; /* List of publication names to subscribe to */
} Subscription;
diff --git a/src/test/subscription/t/022_disable_on_error.pl b/src/test/subscription/t/022_disable_on_error.pl
new file mode 100644
index 0000000000..b084bbab5a
--- /dev/null
+++ b/src/test/subscription/t/022_disable_on_error.pl
@@ -0,0 +1,125 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+
+my @schemas = qw(s1 s2);
+my ($schema, $cmd);
+
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create identical schema, table and index on both the publisher and
+# subscriber, with publications and subscriptions linking the two.
+#
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+for $schema (@schemas)
+{
+ $cmd = qq(
+CREATE SCHEMA $schema;
+CREATE TABLE $schema.tbl (i INT);
+ALTER TABLE $schema.tbl REPLICA IDENTITY FULL;
+CREATE INDEX ${schema}_tbl_idx ON $schema.tbl(i));
+ $node_publisher->safe_psql('postgres', $cmd);
+ $node_subscriber->safe_psql('postgres', $cmd);
+
+ # Create the publication for this table
+ $cmd = qq(
+CREATE PUBLICATION $schema FOR TABLE $schema.tbl);
+ $node_publisher->safe_psql('postgres', $cmd);
+
+ # Create the subscription for this table
+ $cmd = qq(
+CREATE SUBSCRIPTION $schema
+ CONNECTION '$publisher_connstr'
+ PUBLICATION $schema
+ WITH (disable_on_error = true));
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Create an additional unique index in schema s1 on the subscriber only,
+# creating the circumstances for the replication to fail while applying changes
+# to subscription "s1" while succeeding for subscription "s2".
+#
+$cmd = qq(CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+$node_publisher->wait_for_catchup($_) for (@schemas);
+
+# Wait for initial sync to finish as well
+my $synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('s', 'r');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Enter non-unique data for schema "s1" on the publisher. This should cause
+# subscription "s1" on the subscriber to fail and get automatically disabled.
+#
+$cmd = qq(INSERT INTO s1.tbl (i) VALUES (1), (1), (1));
+$node_publisher->safe_psql('postgres', $cmd);
+
+# Wait for subscription "s1" to be automatically disabled.
+#
+my $s1_disabled_query = qq(
+SELECT count(1) = 1
+ FROM pg_catalog.pg_subscription
+ WHERE subname = 's1'
+ AND disabled_by_error IS TRUE);
+$node_subscriber->poll_query_until('postgres', $s1_disabled_query);
+
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+#
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (2));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for subscription "s2". This tests that the
+# problems encountered by subscription "s1" do not cause subscription "s2" to
+# get stuck.
+$node_publisher->wait_for_catchup("s2");
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error
+#
+$cmd = qq(
+SELECT disabled_by_error FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s1 automatically disabled");
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+$cmd = qq(
+SELECT suberrmsg FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ qq(duplicate key value violates unique constraint "s1_tbl_unique"),
+ "subscription s1 disabled by unique key violation");
+
+# Subscription "s2" should still be enabled and have replicated all changes
+#
+$cmd = qq(
+SELECT disabled_by_error FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s2 not automatically disabled");
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = q(SELECT i FROM s2.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "2", "subscription s2 replicated data");
--
2.21.1 (Apple Git-122.3)
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-18 04:47 Amit Kapila <[email protected]>
parent: Mark Dilger <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Amit Kapila @ 2021-06-18 04:47 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; Sawada Masahiko <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Fri, Jun 18, 2021 at 1:48 AM Mark Dilger
<[email protected]> wrote:
>
> Hackers,
>
> Logical replication apply workers for a subscription can easily get stuck in an infinite loop of attempting to apply a change, triggering an error (such as a constraint violation), exiting with an error written to the subscription worker log, and restarting.
>
> As things currently stand, only superusers can create subscriptions. Ongoing work to delegate superuser tasks to non-superusers creates the potential for even more errors to be triggered, specifically, errors where the apply worker does not have permission to make changes to the target table.
>
> The attached patch makes it possible to create a subscription using a new subscription_parameter, "disable_on_error", such that rather than going into an infinite loop, the apply worker will catch errors and automatically disable the subscription, breaking the loop. The new parameter defaults to false. When false, the PG_TRY/PG_CATCH overhead is avoided, so for subscriptions which do not use the feature, there shouldn't be any change. Users can manually clear the error after fixing the underlying issue with an ALTER SUBSCRIPTION .. ENABLE command.
>
I see this idea has merits and it will help users to repair failing
subscriptions. Few points on a quick look at the patch: (a) The patch
seem to be assuming that the error can happen only by the apply worker
but I think the constraint violation can happen via one of the table
sync workers as well, (b) What happens if the error happens when you
are updating the error information in the catalog table. I think
instead of seeing the actual apply time error, the user might see some
other for which it won't be clear what is an appropriate action.
We are also discussing another action like skipping the apply of the
transaction on an error [1]. I think it is better to evaluate both the
proposals as one seems to be an extension of another. Adding
Sawada-San, as he is working on the other proposal.
[1] - https://www.postgresql.org/message-id/CAD21AoDeScrsHhLyEPYqN3sydg6PxAPVBboK%3D30xJfUVihNZDA%40mail.g...
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-18 06:34 Peter Smith <[email protected]>
parent: Mark Dilger <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Peter Smith @ 2021-06-18 06:34 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Fri, Jun 18, 2021 at 6:18 AM Mark Dilger
<[email protected]> wrote:
>
> Hackers,
>
> Logical replication apply workers for a subscription can easily get stuck in an infinite loop of attempting to apply a change, triggering an error (such as a constraint violation), exiting with an error written to the subscription worker log, and restarting.
>
> As things currently stand, only superusers can create subscriptions. Ongoing work to delegate superuser tasks to non-superusers creates the potential for even more errors to be triggered, specifically, errors where the apply worker does not have permission to make changes to the target table.
>
> The attached patch makes it possible to create a subscription using a new subscription_parameter, "disable_on_error", such that rather than going into an infinite loop, the apply worker will catch errors and automatically disable the subscription, breaking the loop. The new parameter defaults to false. When false, the PG_TRY/PG_CATCH overhead is avoided, so for subscriptions which do not use the feature, there shouldn't be any change. Users can manually clear the error after fixing the underlying issue with an ALTER SUBSCRIPTION .. ENABLE command.
>
> In addition to helping on production systems, this makes writing TAP tests involving error conditions simpler. I originally ran into the motivation to write this patch when frustrated that TAP tests needed to parse the apply worker log file to determine whether permission failures were occurring and what they were. It was also obnoxiously easy to have a test get stuck waiting for a permanently stuck subscription to catch up. This helps with both issues.
>
> I don't think this is quite ready for commit, but I'd like feedback if folks like this idea or want to suggest design changes.
I tried your patch.
It applied OK (albeit with whitespace warnings).
The code build and TAP tests are all OK.
Below are a few comments and observations.
COMMENTS
========
(1) PG Docs catalogs.sgml
Documented new column "suberrmsg" but did not document the other new
columns ("disable_on_error", "disabled_by_error")?
------
(2) New column "disabled_by_error".
I wondered if there was actually any need for this column. Isn't the
same information conveyed by just having "subenabled" = false, at same
time as as non-empty "suberrmsg"? This would remove any confusion for
having 2 booleans which both indicate disabled.
------
(3) New columns "disabled_by_error", "disabled_on_error".
All other columns of the pg_subscription have a "sub" prefix.
------
(4) errhint member used?
@@ -91,12 +100,16 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool disable_on_error; /* Whether errors automatically disable */
+ bool disabled_by_error; /* Whether an error has disabled */
bool binary; /* Indicates if the subscription wants data in
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
+ char *errmsg; /* Message from error which disabled */
+ char *errhint; /* Hint from error which disabled */
List *publications; /* List of publication names to subscribe to */
} Subscription;
I did not find any code using that newly added member "errhint".
------
(5) dump.c
i. No mention of new columns "disabled_on_error" and
"disabled_by_error". Is that right?
ii. Shouldn't the code for the "suberrmsg" be qualified with some PG
version number checks?
------
(6) Patch only handles errors only from the Apply worker.
Tablesync can give similar errors (e.g. PK violation during DATASYNC
phase) which will trigger re-launch forever regardless of the setting
of "disabled_on_error".
(confirmed by observations below)
------
(7) TAP test code
+$node_subscriber->init(allows_streaming => 'logical');
AFAIK that "logical" configuration is not necessary for the subscriber side. So,
$node_subscriber->init();
////////////
Some Experiments/Observations
==============================
In general, I found this functionality is useful and it works as
advertised by your patch comment.
======
Test: Display pg_subscription with the new columns
Observation: As expected. But some new colnames are not prefixed like
their peers.
test_sub=# \pset x
Expanded display is on.
test_sub=# select * from pg_subscription;
-[ RECORD 1 ]-----+--------------------------------------------------------
oid | 16394
subdbid | 16384
subname | tap_sub
subowner | 10
subenabled | t
disable_on_error | t
disabled_by_error | f
subbinary | f
substream | f
subconninfo | host=localhost dbname=test_pub application_name=tap_sub
subslotname | tap_sub
subsynccommit | off
suberrmsg |
subpublications | {tap_pub}
======
Test: Cause a PK violation during normal Apply replication (when
"disabled_on_error=true")
Observation: Apply worker stops. Subscription is disabled. Error
message is in the catalog.
2021-06-18 15:12:45.905 AEST [25904] LOG: edata is true for
subscription 'tap_sub': message = "duplicate key value violates unique
constraint "test_tab_pkey"", hint = "<NONE>"
2021-06-18 15:12:45.905 AEST [25904] LOG: logical replication apply
worker for subscription "tap_sub" will stop because the subscription
was disabled due to error
2021-06-18 15:12:45.905 AEST [25904] ERROR: duplicate key value
violates unique constraint "test_tab_pkey"
2021-06-18 15:12:45.905 AEST [25904] DETAIL: Key (a)=(1) already exists.
2021-06-18 15:12:45.908 AEST [19924] LOG: background worker "logical
replication worker" (PID 25904) exited with exit code 1
test_sub=# select * from pg_subscription;
-[ RECORD 1 ]-----+---------------------------------------------------------------
oid | 16394
subdbid | 16384
subname | tap_sub
subowner | 10
subenabled | f
disable_on_error | t
disabled_by_error | t
subbinary | f
substream | f
subconninfo | host=localhost dbname=test_pub application_name=tap_sub
subslotname | tap_sub
subsynccommit | off
suberrmsg | duplicate key value violates unique constraint
"test_tab_pkey"
subpublications | {tap_pub}
======
Test: Try to enable subscription (without fixing the PK violation problem).
Observation. OK. It just stops again
test_sub=# alter subscription tap_sub enable;
ALTER SUBSCRIPTION
test_sub=# 2021-06-18 15:17:18.067 AEST [10228] LOG: logical
replication apply worker for subscription "tap_sub" has started
2021-06-18 15:17:18.078 AEST [10228] LOG: edata is true for
subscription 'tap_sub': message = "duplicate key value violates unique
constraint "test_tab_pkey"", hint = "<NONE>"
2021-06-18 15:17:18.078 AEST [10228] LOG: logical replication apply
worker for subscription "tap_sub" will stop because the subscription
was disabled due to error
2021-06-18 15:17:18.078 AEST [10228] ERROR: duplicate key value
violates unique constraint "test_tab_pkey"
2021-06-18 15:17:18.078 AEST [10228] DETAIL: Key (a)=(1) already exists.
2021-06-18 15:17:18.079 AEST [19924] LOG: background worker "logical
replication worker" (PID 10228) exited with exit code 1
======
Test: Manually disable the subscription (which had previously already
been disabled due to error)
Observation: OK. The suberrmsg gets reset to an empty string.
alter subscription tap_sub disable;
=====
Test: Turn off the disable_on_error
Observation: As expected, now the Apply worker goes into re-launch
forever loop every time it hits PK violation
test_sub=# alter subscription tap_sub set (disable_on_error=false);
ALTER SUBSCRIPTION
...
======
Test: Cause a PK violation in the Tablesync copy (DATASYNC) phase.
(when disable_on_error = true)
Observation: This patch changes nothing for this case. The Tablesyn
re-launchs in a forever loop the same as current functionality.
test_sub=# CREATE SUBSCRIPTION tap_sub CONNECTION 'host=localhost
dbname=test_pub application_name=tap_sub' PUBLICATION tap_pub WITH
(disable_on_error=false);
NOTICE: created replication slot "tap_sub" on publisher
CREATE SUBSCRIPTION
test_sub=# 2021-06-18 15:38:19.547 AEST [18205] LOG: logical
replication apply worker for subscription "tap_sub" has started
2021-06-18 15:38:19.557 AEST [18207] LOG: logical replication table
synchronization worker for subscription "tap_sub", table "test_tab"
has started
2021-06-18 15:38:19.610 AEST [18207] ERROR: duplicate key value
violates unique constraint "test_tab_pkey"
2021-06-18 15:38:19.610 AEST [18207] DETAIL: Key (a)=(1) already exists.
2021-06-18 15:38:19.610 AEST [18207] CONTEXT: COPY test_tab, line 1
2021-06-18 15:38:19.611 AEST [19924] LOG: background worker "logical
replication worker" (PID 18207) exited with exit code 1
2021-06-18 15:38:24.634 AEST [18369] LOG: logical replication table
synchronization worker for subscription "tap_sub", table "test_tab"
has started
2021-06-18 15:38:24.689 AEST [18369] ERROR: duplicate key value
violates unique constraint "test_tab_pkey"
2021-06-18 15:38:24.689 AEST [18369] DETAIL: Key (a)=(1) already exists.
2021-06-18 15:38:24.689 AEST [18369] CONTEXT: COPY test_tab, line 1
2021-06-18 15:38:24.690 AEST [19924] LOG: background worker "logical
replication worker" (PID 18369) exited with exit code 1
2021-06-18 15:38:29.701 AEST [18521] LOG: logical replication table
synchronization worker for subscription "tap_sub", table "test_tab"
has started
2021-06-18 15:38:29.765 AEST [18521] ERROR: duplicate key value
violates unique constraint "test_tab_pkey"
2021-06-18 15:38:29.765 AEST [18521] DETAIL: Key (a)=(1) already exists.
2021-06-18 15:38:29.765 AEST [18521] CONTEXT: COPY test_tab, line 1
2021-06-18 15:38:29.766 AEST [19924] LOG: background worker "logical
replication worker" (PID 18521) exited with exit code 1
etc...
-[ RECORD 1 ]-----+--------------------------------------------------------
oid | 16399
subdbid | 16384
subname | tap_sub
subowner | 10
subenabled | t
disable_on_error | f
disabled_by_error | f
subbinary | f
substream | f
subconninfo | host=localhost dbname=test_pub application_name=tap_sub
subslotname | tap_sub
subsynccommit | off
suberrmsg |
subpublications | {tap_pub}
------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-18 19:36 Mark Dilger <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Mark Dilger @ 2021-06-18 19:36 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Sawada Masahiko <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Jun 17, 2021, at 9:47 PM, Amit Kapila <[email protected]> wrote:
>
> (a) The patch
> seem to be assuming that the error can happen only by the apply worker
> but I think the constraint violation can happen via one of the table
> sync workers as well
You are right. Peter mentioned the same thing, and it is clearly so. I am working to repair this fault in v2 of the patch.
> (b) What happens if the error happens when you
> are updating the error information in the catalog table.
I think that is an entirely different kind of error. The patch attempts to catch errors caused by the user, not by core functionality of the system failing. If there is a fault that prevents the catalogs from being updated, it is unclear what the patch can do about that.
> I think
> instead of seeing the actual apply time error, the user might see some
> other for which it won't be clear what is an appropriate action.
Good point.
Before trying to do much of anything with the caught error, the v2 patch logs the error. If the subsequent efforts to disable the subscription fail, at least the logs should contain the initial failure message. The v1 patch emitted a log message much further down, and really just intended for debugging the patch itself, with many opportunities for something else to throw before the log is written.
> We are also discussing another action like skipping the apply of the
> transaction on an error [1]. I think it is better to evaluate both the
> proposals as one seems to be an extension of another.
Thanks for the link.
I think they are two separate options. For some users and data patterns, subscriber-side skipping of specific problematic commits will be fine. For other usage patterns, skipping earlier commits will results in more and more data integrity problems (foreign key references, etc.) such that the failures will snowball with skipping becoming the norm rather than the exception. Users with those usage patterns would likely prefer the subscription to automatically be disabled until manual intervention can clean up the problem.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-19 00:03 Mark Dilger <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 0 replies; 74+ messages in thread
From: Mark Dilger @ 2021-06-19 00:03 UTC (permalink / raw)
To: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
> On Jun 17, 2021, at 11:34 PM, Peter Smith <[email protected]> wrote:
>
> I tried your patch.
Thanks for the quick and thorough review!
> (2) New column "disabled_by_error".
>
> I wondered if there was actually any need for this column. Isn't the
> same information conveyed by just having "subenabled" = false, at same
> time as as non-empty "suberrmsg"? This would remove any confusion for
> having 2 booleans which both indicate disabled.
Yeah, I wondered about that before posting v1. I removed the disabled_by_error field for v2.
> (3) New columns "disabled_by_error", "disabled_on_error".
>
> All other columns of the pg_subscription have a "sub" prefix.
I don't feel strongly about this. How about "subdisableonerr"? I used that in v2.
> I did not find any code using that newly added member "errhint".
Thanks for catching that. I had tried to remove all references to "errhint" before posting v1. The original idea was that both the message and hint of the error would be kept, but in testing I found the hint field was typically empty, so I removed it. Sorry that I left one mention of it lying around.
> (5) dump.c
I didn't bother getting pg_dump working before posting v1, and I still have not done so, as I mainly want to solicit feedback on whether the basic direction I am going will work for the community.
> (6) Patch only handles errors only from the Apply worker.
>
> Tablesync can give similar errors (e.g. PK violation during DATASYNC
> phase) which will trigger re-launch forever regardless of the setting
> of "disabled_on_error".
> (confirmed by observations below)
Yes, this is a good point, and also mentioned by Amit. I have fixed it in v2 and adjusted the regression test to trigger an automatic disabling for initial table sync as well as for change replication.
> 2021-06-18 15:12:45.905 AEST [25904] LOG: edata is true for
> subscription 'tap_sub': message = "duplicate key value violates unique
> constraint "test_tab_pkey"", hint = "<NONE>"
You didn't call this out, but FYI, I don't intend to leave this particular log message in the patch. It was for development only. I have removed it for v2 and have added a different log message much sooner after catching the error, to avoid squashing the error in case some other action fails.
The regression test shows this, if you open tmp_check/log/022_disable_on_error_subscriber.log:
2021-06-18 16:25:20.138 PDT [56926] LOG: logical replication subscription "s1" will be disabled due to error: duplicate key value violates unique constraint "s1_tbl_unique"
2021-06-18 16:25:20.139 PDT [56926] ERROR: duplicate key value violates unique constraint "s1_tbl_unique"
2021-06-18 16:25:20.139 PDT [56926] DETAIL: Key (i)=(1) already exists.
2021-06-18 16:25:20.139 PDT [56926] CONTEXT: COPY tbl, line 2
The first line logs the error prior to attempting to disable the subscription, and the next three lines are due to rethrowing the error after committing the successful disabling of the subscription. If the attempt to disable the subscription itself throws, these additional three lines won't show up, but the first one should. Amit mentioned this upthread. Do you think this will be ok, or would you like to also have a suberrdetail field so that the detail doesn't get lost? I haven't added such an extra field, and am inclined to think it would be excessive, but maybe others feel differently?
> ======
>
> Test: Cause a PK violation in the Tablesync copy (DATASYNC) phase.
> (when disable_on_error = true)
> Observation: This patch changes nothing for this case. The Tablesyn
> re-launchs in a forever loop the same as current functionality.
In v2, tablesync copy errors should also be caught. The test has been extended to cover this also.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
[application/octet-stream] v2-0001-Optionally-disabling-subscriptions-on-error.patch (31.3K, ../../[email protected]/2-v2-0001-Optionally-disabling-subscriptions-on-error.patch)
download | inline diff:
From 84cdb2bf2aef276255971be47cf48c6dde7e0564 Mon Sep 17 00:00:00 2001
From: Mark Dilger <[email protected]>
Date: Thu, 17 Jun 2021 12:39:19 -0700
Subject: [PATCH v2] Optionally disabling subscriptions on error
Logical replication apply workers for a subscription can easily get
stuck in an infinite loop of attempting to apply a change,
triggering an error (such as a constraint violation), exiting with
an error written to the subscription worker log, and restarting.
To partially remedy the situation, adding a new
subscription_parameter named 'disable_on_error'. To be consistent
with old behavior, the parameter defaults to false. When true, the
apply worker catches errors thrown, and for errors that are deemed
not to be transient, disables the subscription in order to break the
loop. A new column in the pg_subscription table helps diagnose the
situation: when this has occurred, 'suberrmsg' includes the message
field of the error. The error is still also written to the logs.
In addition to helping on production systems, this makes writing TAP
tests involving error conditions simpler. Rather than having to
open and parse the apply worker's log file, the test can query the
pg_subscription table. It also helps that the workers don't go into
an infinite loop during the test.
---
doc/src/sgml/catalogs.sgml | 10 +
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 9 +
src/backend/catalog/system_views.sql | 8 +-
src/backend/commands/subscriptioncmds.c | 57 +++++
src/backend/replication/logical/launcher.c | 1 +
src/backend/replication/logical/worker.c | 189 +++++++++++++++-
src/bin/pg_dump/pg_dump.c | 6 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/include/catalog/pg_subscription.h | 8 +
src/test/perl/PostgresNode.pm | 40 ++++
.../subscription/t/022_disable_on_error.pl | 205 ++++++++++++++++++
12 files changed, 540 insertions(+), 6 deletions(-)
create mode 100644 src/test/subscription/t/022_disable_on_error.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index f517a7d4af..6fd95cb7ca 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7671,6 +7671,16 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>suberrmsg</structfield> <type>text</type>
+ </para>
+ <para>
+ The message from the error which disabled the subscription, if it has
+ been automatically disabled.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subpublications</structfield> <type>text[]</type>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index e812beee37..6ec6524901 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -117,6 +117,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>disable_on_error</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription should be automatically disabled
+ if replicating data from the publisher triggers non-transient errors
+ such as referential integrity or permissions errors. The default is
+ <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>enabled</literal> (<type>boolean</type>)</term>
<listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 29fc4218cd..5781da062c 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->binary = subform->subbinary;
sub->stream = subform->substream;
@@ -95,6 +96,14 @@ GetSubscription(Oid subid, bool missing_ok)
Assert(!isnull);
sub->synccommit = TextDatumGetCString(datum);
+ /* Get errmsg */
+ datum = SysCacheGetAttr(SUBSCRIPTIONOID,
+ tup,
+ Anum_pg_subscription_suberrmsg,
+ &isnull);
+ Assert(!isnull);
+ sub->errmsg = TextDatumGetCString(datum);
+
/* Get publications */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
tup,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 999d984068..0e840c1a34 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1252,8 +1252,10 @@ CREATE VIEW pg_replication_origin_status AS
REVOKE ALL ON pg_replication_origin_status FROM public;
--- All columns of pg_subscription except subconninfo are publicly readable.
+-- All columns of pg_subscription except subconninfo and suberrmsg are publicly
+-- readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
- substream, subslotname, subsynccommit, subpublications)
+GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subdisableonerr,
+ subbinary, substream, subslotname, subsynccommit,
+ subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 75e195f286..9f8adf529c 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -63,6 +63,7 @@ static void
parse_subscription_options(List *options,
bool *connect,
bool *enabled_given, bool *enabled,
+ bool *disableonerr_given, bool *disableonerr,
bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data,
@@ -84,13 +85,24 @@ parse_subscription_options(List *options,
*connect = true;
if (enabled)
{
+ Assert(enabled_given);
+
*enabled_given = false;
*enabled = true;
}
+ if (disableonerr)
+ {
+ Assert(disableonerr_given);
+
+ *disableonerr_given = false;
+ *disableonerr = false;
+ }
if (create_slot)
*create_slot = true;
if (slot_name)
{
+ Assert(slot_name_given);
+
*slot_name_given = false;
*slot_name = NULL;
}
@@ -102,11 +114,15 @@ parse_subscription_options(List *options,
*refresh = true;
if (binary)
{
+ Assert(binary_given);
+
*binary_given = false;
*binary = false;
}
if (streaming)
{
+ Assert(streaming_given);
+
*streaming_given = false;
*streaming = false;
}
@@ -136,6 +152,16 @@ parse_subscription_options(List *options,
*enabled_given = true;
*enabled = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "disable_on_error") == 0 &&
+ disableonerr)
+ {
+ if (*disableonerr_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ *disableonerr_given = true;
+ *disableonerr = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "create_slot") == 0 && create_slot)
{
if (create_slot_given)
@@ -334,6 +360,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool connect;
bool enabled_given;
bool enabled;
+ bool disableonerr_given;
+ bool disableonerr;
bool copy_data;
bool streaming;
bool streaming_given;
@@ -355,6 +383,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options,
&connect,
&enabled_given, &enabled,
+ &disableonerr_given,
+ &disableonerr,
&create_slot,
&slotname_given, &slotname,
©_data,
@@ -427,6 +457,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_subdisableonerr - 1] =
+ BoolGetDatum(disableonerr);
values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_substream - 1] = BoolGetDatum(streaming);
values[Anum_pg_subscription_subconninfo - 1] =
@@ -438,6 +470,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
nulls[Anum_pg_subscription_subslotname - 1] = true;
values[Anum_pg_subscription_subsynccommit - 1] =
CStringGetTextDatum(synchronous_commit);
+ values[Anum_pg_subscription_suberrmsg - 1] =
+ CStringGetTextDatum("");
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
@@ -799,6 +833,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
{
case ALTER_SUBSCRIPTION_OPTIONS:
{
+ bool disableonerr;
+ bool disableonerr_given;
char *slotname;
bool slotname_given;
char *synchronous_commit;
@@ -810,6 +846,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options,
NULL, /* no "connect" */
NULL, NULL, /* no "enabled" */
+ &disableonerr_given,
+ &disableonerr,
NULL, /* no "create_slot" */
&slotname_given, &slotname,
NULL, /* no "copy_data" */
@@ -818,6 +856,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
&binary_given, &binary,
&streaming_given, &streaming);
+ if (disableonerr_given)
+ {
+ values[Anum_pg_subscription_subdisableonerr - 1] =
+ BoolGetDatum(disableonerr);
+ replaces[Anum_pg_subscription_subdisableonerr - 1 ] =
+ true;
+ }
+
if (slotname_given)
{
if (sub->enabled && !slotname)
@@ -867,6 +913,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options,
NULL, /* no "connect" */
&enabled_given, &enabled,
+ NULL, NULL, /* no "disableonerr" */
NULL, /* no "create_slot" */
NULL, NULL, /* no "slot_name" */
NULL, /* no "copy_data" */
@@ -885,6 +932,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
BoolGetDatum(enabled);
replaces[Anum_pg_subscription_subenabled - 1] = true;
+ /*
+ * Manually enabling or disabling a subscription clears any
+ * error which automatically disabled it.
+ */
+ values[Anum_pg_subscription_suberrmsg - 1] = CStringGetTextDatum("");
+ replaces[Anum_pg_subscription_suberrmsg - 1] = true;
+
if (enabled)
ApplyLauncherWakeupAtCommit();
@@ -912,6 +966,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options,
NULL, /* no "connect" */
NULL, NULL, /* no "enabled" */
+ NULL, NULL, /* no "disableonerr" */
NULL, /* no "create_slot" */
NULL, NULL, /* no "slot_name" */
©_data,
@@ -958,6 +1013,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options,
NULL, /* no "connect" */
NULL, NULL, /* no "enabled" */
+ NULL, NULL, /* no "disableonerr" */
NULL, /* no "create_slot" */
NULL, NULL, /* no "slot_name" */
isadd ? ©_data : NULL, /* for drop, no
@@ -1005,6 +1061,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options,
NULL, /* no "connect" */
NULL, NULL, /* no "enabled" */
+ NULL, NULL, /* no "disableonerr" */
NULL, /* no "create_slot" */
NULL, NULL, /* no "slot_name" */
©_data,
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index e3b11daa89..e7279ea86b 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -132,6 +132,7 @@ get_subscription_list(void)
sub->dbid = subform->subdbid;
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->name = pstrdup(NameStr(subform->subname));
/* We don't fill fields we are not interested in. */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index bbb659dad0..54fd61afe5 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -62,6 +62,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/partition.h"
#include "catalog/pg_inherits.h"
@@ -2412,6 +2413,160 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
walrcv_endstreaming(LogRepWorkerWalRcvConn, &tli);
}
+/*
+ * Errors which are transient, network protocol related, or resource exhaustion
+ * related, should not disable a subscription. These may clear up without user
+ * intervention in the subscription, schema, or data being replicated.
+ */
+static bool
+IsSubscriptionDisablingError(void)
+{
+ switch (geterrcode())
+ {
+ case ERRCODE_CONNECTION_EXCEPTION:
+ case ERRCODE_CONNECTION_DOES_NOT_EXIST:
+ case ERRCODE_CONNECTION_FAILURE:
+ case ERRCODE_SQLCLIENT_UNABLE_TO_ESTABLISH_SQLCONNECTION:
+ case ERRCODE_SQLSERVER_REJECTED_ESTABLISHMENT_OF_SQLCONNECTION:
+ case ERRCODE_TRANSACTION_RESOLUTION_UNKNOWN:
+ case ERRCODE_PROTOCOL_VIOLATION:
+ case ERRCODE_INSUFFICIENT_RESOURCES:
+ case ERRCODE_DISK_FULL:
+ case ERRCODE_OUT_OF_MEMORY:
+ case ERRCODE_TOO_MANY_CONNECTIONS:
+ case ERRCODE_CONFIGURATION_LIMIT_EXCEEDED:
+ case ERRCODE_PROGRAM_LIMIT_EXCEEDED:
+ case ERRCODE_STATEMENT_TOO_COMPLEX:
+ case ERRCODE_TOO_MANY_COLUMNS:
+ case ERRCODE_TOO_MANY_ARGUMENTS:
+ case ERRCODE_OPERATOR_INTERVENTION:
+ case ERRCODE_QUERY_CANCELED:
+ case ERRCODE_ADMIN_SHUTDOWN:
+ case ERRCODE_CRASH_SHUTDOWN:
+ case ERRCODE_CANNOT_CONNECT_NOW:
+ case ERRCODE_DATABASE_DROPPED:
+ case ERRCODE_IDLE_SESSION_TIMEOUT:
+ return false;
+ default:
+ break;
+ }
+
+ return true;
+}
+
+/*
+ * Recover from a possibly aborted transaction state and disable the current
+ * subscription.
+ */
+static ErrorData *
+DisableSubscriptionOnError(MemoryContext mcxt)
+{
+ Relation rel;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Oid subid;
+ Form_pg_subscription subform;
+ ErrorData *edata;
+
+ /*
+ * Clean up from the error and get a fresh transaction in which to
+ * disable the subscription.
+ */
+ MemoryContextSwitchTo(mcxt);
+ edata = CopyErrorData();
+
+ ereport(LOG,
+ (errmsg("logical replication subscription \"%s\" will be disabled due to error: %s",
+ MySubscription->name, edata->message)));
+
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+
+ StartTransactionCommand();
+
+ /* Look up our subscription in the catalogs */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy2(SUBSCRIPTIONNAME, MyDatabaseId,
+ CStringGetDatum(MySubscription->name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("subscription \"%s\" does not exist",
+ MySubscription->name)));
+
+ subform = (Form_pg_subscription) GETSTRUCT(tup);
+ subid = subform->oid;
+ LockSharedObject(SubscriptionRelationId, subid, 0, AccessExclusiveLock);
+
+ /*
+ * We would not be here unless this subscription's disableonerr
+ * field was true when our worker began applying changes, but check
+ * whether that field has changed in the interim.
+ */
+ if (!subform->subdisableonerr)
+ ReThrowError(edata);
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Set the subscription to disabled, and note the reason. */
+ values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(false);
+ replaces[Anum_pg_subscription_subenabled - 1] = true;
+ values[Anum_pg_subscription_suberrmsg - 1] =
+ CStringGetTextDatum(edata->message ? edata->message : "");
+ replaces[Anum_pg_subscription_suberrmsg - 1] = true;
+
+ /* Update the catalog */
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+ heap_freetuple(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ CommitTransactionCommand();
+
+ return edata;
+}
+
+/*
+ * Apply main loop with logic to disable a stuck subscription
+ *
+ * Applies changes, and for non-transient errors, catches the error and
+ * disables the subscription before rethrowing.
+ */
+static void
+LogicalRepApplyLoopTry(XLogRecPtr last_received)
+{
+ MemoryContext mcxt = CurrentMemoryContext;
+ bool did_error = false;
+
+ PG_TRY();
+ {
+ LogicalRepApplyLoop(last_received);
+ }
+ PG_CATCH();
+ {
+ if (IsSubscriptionDisablingError())
+ did_error = true;
+ else
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /*
+ * If we caught an error above, disable the subscription and record copies
+ * of the relevant error text, then rethrow the error we caught so it gets
+ * logged and our process exits appropriately.
+ */
+ if (did_error)
+ ReThrowError(DisableSubscriptionOnError(mcxt));
+}
+
/*
* Send a Standby Status Update message to server.
*
@@ -3167,7 +3322,34 @@ ApplyWorkerMain(Datum main_arg)
char *syncslotname;
/* This is table synchronization worker, call initial sync. */
- syncslotname = LogicalRepSyncTableStart(&origin_startpos);
+ if (MySubscription->disableonerr)
+ {
+ MemoryContext mcxt = CurrentMemoryContext;
+ bool did_error = false;
+
+ PG_TRY();
+ {
+ syncslotname = LogicalRepSyncTableStart(&origin_startpos);
+ }
+ PG_CATCH();
+ {
+ if (IsSubscriptionDisablingError())
+ did_error = true;
+ else
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /*
+ * If we caught an error above, disable the subscription and record copies
+ * of the relevant error text, then rethrow the error we caught so it gets
+ * logged and our process exits appropriately.
+ */
+ if (did_error)
+ ReThrowError(DisableSubscriptionOnError(mcxt));
+ }
+ else
+ syncslotname = LogicalRepSyncTableStart(&origin_startpos);
/* allocate slot name in long-lived context */
myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
@@ -3241,7 +3423,10 @@ ApplyWorkerMain(Datum main_arg)
walrcv_startstreaming(LogRepWorkerWalRcvConn, &options);
/* Run the main loop. */
- LogicalRepApplyLoop(origin_startpos);
+ if (MySubscription->disableonerr)
+ LogicalRepApplyLoopTry(origin_startpos);
+ else
+ LogicalRepApplyLoop(origin_startpos);
proc_exit(0);
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 8f53cc7c3b..16d9d006d1 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4307,6 +4307,7 @@ getSubscriptions(Archive *fout)
int i_subconninfo;
int i_subslotname;
int i_subsynccommit;
+ int i_suberrmsg;
int i_subpublications;
int i_subbinary;
int i,
@@ -4338,7 +4339,7 @@ getSubscriptions(Archive *fout)
"SELECT s.tableoid, s.oid, s.subname,\n"
" (%s s.subowner) AS rolname,\n"
" s.subconninfo, s.subslotname, s.subsynccommit,\n"
- " s.subpublications,\n",
+ " s.suberrmsg, s.subpublications,\n",
username_subquery);
if (fout->remoteVersion >= 140000)
@@ -4367,6 +4368,7 @@ getSubscriptions(Archive *fout)
i_subconninfo = PQfnumber(res, "subconninfo");
i_subslotname = PQfnumber(res, "subslotname");
i_subsynccommit = PQfnumber(res, "subsynccommit");
+ i_suberrmsg = PQfnumber(res, "suberrmsg");
i_subpublications = PQfnumber(res, "subpublications");
i_subbinary = PQfnumber(res, "subbinary");
i_substream = PQfnumber(res, "substream");
@@ -4389,6 +4391,8 @@ getSubscriptions(Archive *fout)
subinfo[i].subslotname = pg_strdup(PQgetvalue(res, i, i_subslotname));
subinfo[i].subsynccommit =
pg_strdup(PQgetvalue(res, i, i_subsynccommit));
+ subinfo[i].suberrmsg =
+ pg_strdup(PQgetvalue(res, i, i_suberrmsg));
subinfo[i].subpublications =
pg_strdup(PQgetvalue(res, i, i_subpublications));
subinfo[i].subbinary =
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 49e1b0a09c..3c2a8dd38c 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -638,6 +638,7 @@ typedef struct _SubscriptionInfo
char *subbinary;
char *substream;
char *subsynccommit;
+ char *suberrmsg;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0060ebfb40..b094bd082f 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -52,6 +52,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subdisableonerr; /* True if apply errors should
+ * disable the subscription upon error */
+
bool subbinary; /* True if the subscription wants the
* publisher to send data in binary */
@@ -67,6 +70,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
/* Synchronous commit setting for worker */
text subsynccommit BKI_FORCE_NOT_NULL;
+ /* Message from error which disabled this subscription */
+ text suberrmsg BKI_FORCE_NOT_NULL;
+
/* List of publications subscribed to */
text subpublications[1] BKI_FORCE_NOT_NULL;
#endif
@@ -91,12 +97,14 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool disableonerr; /* Whether errors automatically disable */
bool binary; /* Indicates if the subscription wants data in
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
+ char *errmsg; /* Message from error which disabled */
List *publications; /* List of publication names to subscribe to */
} Subscription;
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 2027cbf43d..33dc9d5367 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -2442,6 +2442,46 @@ sub wait_for_slot_catchup
return;
}
+=pot
+
+=item $node->wait_for_subscription($dbname, @subcriptions)
+
+Wait for the named subscriptions to catch up or to be disabled.
+
+=cut
+
+sub wait_for_subscriptions
+{
+ my ($self, $dbname, @subscriptions) = @_;
+
+ # Unique-ify the subscriptions passed by the caller
+ my %unique = map { $_ => 1 } @subscriptions;
+ my @unique = sort keys %unique;
+ my $unique_count = scalar(@unique);
+
+ # Construct a SQL list from the unique subscription names
+ my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
+ my $sublist = join(', ', map { "'$_'" } @escaped);
+
+ my $polling_sql = qq(
+ SELECT COUNT(1) = $unique_count FROM
+ (SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ LEFT JOIN pg_catalog.pg_subscription_rel sr
+ ON sr.srsubid = s.oid
+ WHERE (sr IS NULL OR sr.srsubstate IN ('s', 'r'))
+ AND s.subname IN ($sublist)
+ AND s.subenabled IS TRUE
+ UNION
+ SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ WHERE s.subname IN ($sublist)
+ AND s.subenabled IS FALSE
+ ) AS synced_or_disabled
+ );
+ return $self->poll_query_until($dbname, $polling_sql);
+}
+
=pod
=item $node->query_hash($dbname, $query, @columns)
diff --git a/src/test/subscription/t/022_disable_on_error.pl b/src/test/subscription/t/022_disable_on_error.pl
new file mode 100644
index 0000000000..5c10981a6b
--- /dev/null
+++ b/src/test/subscription/t/022_disable_on_error.pl
@@ -0,0 +1,205 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 13;
+
+my @schemas = qw(s1 s2);
+my ($schema, $cmd);
+
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+# Create identical schema, table and index on both the publisher and
+# subscriber
+#
+for $schema (@schemas)
+{
+ $cmd = qq(
+CREATE SCHEMA $schema;
+CREATE TABLE $schema.tbl (i INT);
+ALTER TABLE $schema.tbl REPLICA IDENTITY FULL;
+CREATE INDEX ${schema}_tbl_idx ON $schema.tbl(i));
+ $node_publisher->safe_psql('postgres', $cmd);
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Create non-unique data in both schemas on the publisher.
+#
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (1), (1), (1));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Create an additional unique index in schema s1 on the subscriber only. When
+# we create subscriptions, below, this should cause subscription "s1" on the
+# subscriber to fail during initial synchronization and to get automatically
+# disabled.
+#
+$cmd = qq(CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Create publications and subscriptions linking the schemas on
+# the publisher with those on the subscriber. This tests that the
+# uniqueness violations cause subscription "s1" to fail during
+# initial synchronization.
+#
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+for $schema (@schemas)
+{
+ # Create the publication for this table
+ $cmd = qq(
+CREATE PUBLICATION $schema FOR TABLE $schema.tbl);
+ $node_publisher->safe_psql('postgres', $cmd);
+
+ # Create the subscription for this table
+ $cmd = qq(
+CREATE SUBSCRIPTION $schema
+ CONNECTION '$publisher_connstr'
+ PUBLICATION $schema
+ WITH (disable_on_error = true));
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Wait for the initial subscription synchronizations to finish or fail.
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+$cmd = qq(
+SELECT suberrmsg FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ qq(duplicate key value violates unique constraint "s1_tbl_unique"),
+ "subscription s1 disabled during synchronization by unique key violation");
+
+# Subscription "s2" should have copied the initial data without incident.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT i, COUNT(*) FROM s2.tbl GROUP BY i);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "1|3",
+ "subscription s2 replicated initial data");
+
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+#
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (2));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that the
+# problems encountered by subscription "s1" do not cause subscription "s2" to
+# get stuck.
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should still be disabled and have replicated no data
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 still disabled");
+$cmd = qq(
+SELECT suberrmsg FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ qq(duplicate key value violates unique constraint "s1_tbl_unique"),
+ "subscription s1 still disabled by unique key violation");
+
+# Subscription "s2" should still be enabled and have replicated all changes
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s2 replicated data");
+
+# Drop the unique index on "s1" which caused the subscription to be disabled
+#
+$cmd = qq(DROP INDEX s1.s1_tbl_unique);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Re-enable the subscription "s1"
+#
+$cmd = q(ALTER SUBSCRIPTION s1 ENABLE);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Wait for the data to replicate
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check that we have the new data in s1.tbl
+#
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s1.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s1 replicated data");
+
+# Delete the data from the subscriber only, and recreate the unique index
+#
+$cmd = q(
+DELETE FROM s1.tbl;
+CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Add more non-unique data to the publisher
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (3), (3), (3));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that
+# uniqueness violations encountered during replication cause s1 to be disabled.
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+$cmd = qq(
+SELECT suberrmsg FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ qq(duplicate key value violates unique constraint "s1_tbl_unique"),
+ "subscription s1 disabled during replication by unique key violation");
+
+# Subscription "s2" should have copied the initial data without incident.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "3|7",
+ "subscription s2 replicated additional data");
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
2.21.1 (Apple Git-122.3)
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-19 10:17 Amit Kapila <[email protected]>
parent: Mark Dilger <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Amit Kapila @ 2021-06-19 10:17 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Sawada Masahiko <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sat, Jun 19, 2021 at 1:06 AM Mark Dilger
<[email protected]> wrote:
> > On Jun 17, 2021, at 9:47 PM, Amit Kapila <[email protected]> wrote:
>
> > We are also discussing another action like skipping the apply of the
> > transaction on an error [1]. I think it is better to evaluate both the
> > proposals as one seems to be an extension of another.
>
> Thanks for the link.
>
> I think they are two separate options.
>
Right, but there are things that could be common from the design
perspective. For example, why is it mandatory to update this conflict
( error) information in the system catalog instead of displaying it
via some stats view? Also, why not also log the xid of the failed
transaction?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-19 14:44 Mark Dilger <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 3 replies; 74+ messages in thread
From: Mark Dilger @ 2021-06-19 14:44 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Sawada Masahiko <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Jun 19, 2021, at 3:17 AM, Amit Kapila <[email protected]> wrote:
>
> Right, but there are things that could be common from the design
> perspective.
I went to reconcile my patch with that from [1] only to discover there is no patch on that thread. Is there one in progress that I can see?
I don't mind trying to reconcile this patch with what you're discussing in [1], but I am a bit skeptical about [1] becoming a reality and I don't want to entirely hitch this patch to that effort. This can be committed with or without any solution to the idea in [1]. The original motivation for this patch was that the TAP tests don't have a great way to deal with a subscription getting into a fail-retry infinite loop, which makes it harder for me to make progress on [2]. That doesn't absolve me of the responsibility of making this patch a good one, but it does motivate me to keep it simple.
> For example, why is it mandatory to update this conflict
> ( error) information in the system catalog instead of displaying it
> via some stats view?
The catalog must be updated to disable the subscription, so placing the error information in the same row doesn't require any independent touching of the catalogs. Likewise, the catalog must be updated to re-enable the subscription, so clearing the error from that same row doesn't require any independent touching of the catalogs.
The error information does not *need* to be included in the catalog, but placing the information in any location that won't survive server restart leaves the user no information about why the subscription got disabled after a restart (or crash + restart) happens.
Furthermore, since v2 removed the "disabled_by_error" field in favor of just using subenabled + suberrmsg to determine if the subscription was automatically disabled, not having the information in the catalog would make it ambiguous whether the subscription was manually or automatically disabled.
> Also, why not also log the xid of the failed
> transaction?
We could also do that. Reading [1], it seems you are overly focused on user-facing xids. The errdetail in the examples I've been using for testing, and the one mentioned in [1], contain information about the conflicting data. I think users are more likely to understand that a particular primary key value cannot be replicated because it is not unique than to understand that a particular xid cannot be replicated. (Likewise for permissions errors.) For example:
2021-06-18 16:25:20.139 PDT [56926] ERROR: duplicate key value violates unique constraint "s1_tbl_unique"
2021-06-18 16:25:20.139 PDT [56926] DETAIL: Key (i)=(1) already exists.
2021-06-18 16:25:20.139 PDT [56926] CONTEXT: COPY tbl, line 2
This tells the user what they need to clean up before they can continue. Telling them which xid tried to apply the change, but not the change itself or the conflict itself, seems rather unhelpful. So at best, the xid is an additional piece of information. I'd rather have both the ERROR and DETAIL fields above and not the xid than have the xid and lack one of those two fields. Even so, I have not yet included the DETAIL field because I didn't want to bloat the catalog.
For the problem in [1], having the xid is more important than it is in my patch, because the user is expected in [1] to use the xid as a handle. But that seems like an odd interface to me. Imagine that a transaction on the publisher side inserted a batch of data, and only a subset of that data conflicts on the subscriber side. What advantage is there in skipping the entire transaction? Wouldn't the user rather skip just the problematic rows? I understand that on the subscriber side it is difficult to do so, but if you are going to implement this sort of thing, it makes more sense to allow the user to filter out data that is problematic rather than filtering out xids that are problematic, and the filter shouldn't just be an in-or-out filter, but rather a mapping function that can redirect the data someplace else or rewrite it before inserting or change the pre-existing conflicting data prior to applying the problematic data or whatever. That's a huge effort, of course, but if the idea in [1] goes in that direction, I don't want my patch to have already added an xid field which ultimately nobody wants.
[1] - https://www.postgresql.org/message-id/CAD21AoDeScrsHhLyEPYqN3sydg6PxAPVBboK%3D30xJfUVihNZDA%40mail.g...
[2] - https://www.postgresql.org/message-id/flat/915B995D-1D79-4E0A-BD8D-3B267925FCE9%40enterprisedb.com#d...
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-19 16:21 Mark Dilger <[email protected]>
parent: Mark Dilger <[email protected]>
2 siblings, 0 replies; 74+ messages in thread
From: Mark Dilger @ 2021-06-19 16:21 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Sawada Masahiko <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Jun 19, 2021, at 7:44 AM, Mark Dilger <[email protected]> wrote:
>
> Wouldn't the user rather skip just the problematic rows? I understand that on the subscriber side it is difficult to do so, but if you are going to implement this sort of thing, it makes more sense to allow the user to filter out data that is problematic rather than filtering out xids that are problematic, and the filter shouldn't just be an in-or-out filter, but rather a mapping function that can redirect the data someplace else or rewrite it before inserting or change the pre-existing conflicting data prior to applying the problematic data or whatever.
Thinking about this some more, it seems my patch already sets the stage for this sort of thing.
We could extend the concept of triggers to something like ErrorTriggers that could be associated with subscriptions. I already have the code catching errors for subscriptions where disable_on_error is true. We could use that same code path for subscriptions that have one or more BEFORE or AFTER ErrorTriggers defined. We could pass the trigger all the error context information along with the row and subscription information, and allow the trigger to either modify the data being replicated or make modifications to the table being changed. I think having support for both BEFORE and AFTER would be important, as a common design pattern might be to move aside the conflicting rows in the BEFORE trigger, then reconcile and merge them back into the table in the AFTER trigger. If the xid still cannot be replicated after one attempt using the triggers, the second attempt to disable the subscription instead.
There are a lot of details to consider, but to my mind this idea is much more user friendly than the idea that users should muck about with xids for arbitrarily many conflicting transactions.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-21 02:17 Masahiko Sawada <[email protected]>
parent: Mark Dilger <[email protected]>
2 siblings, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2021-06-21 02:17 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sat, Jun 19, 2021 at 11:44 PM Mark Dilger
<[email protected]> wrote:
>
>
>
> > On Jun 19, 2021, at 3:17 AM, Amit Kapila <[email protected]> wrote:
> >
> > Right, but there are things that could be common from the design
> > perspective.
>
> I went to reconcile my patch with that from [1] only to discover there is no patch on that thread. Is there one in progress that I can see?
I will submit the patch.
>
> I don't mind trying to reconcile this patch with what you're discussing in [1], but I am a bit skeptical about [1] becoming a reality and I don't want to entirely hitch this patch to that effort. This can be committed with or without any solution to the idea in [1]. The original motivation for this patch was that the TAP tests don't have a great way to deal with a subscription getting into a fail-retry infinite loop, which makes it harder for me to make progress on [2]. That doesn't absolve me of the responsibility of making this patch a good one, but it does motivate me to keep it simple.
There was a discussion that the skipping transaction patch would also
need to have a feature that tells users the details of the last
failure transaction such as its XID, timestamp, action etc. In that
sense, those two patches might need the common infrastructure that the
apply workers leave the error details somewhere so that the users can
see it.
> > For example, why is it mandatory to update this conflict
> > ( error) information in the system catalog instead of displaying it
> > via some stats view?
>
> The catalog must be updated to disable the subscription, so placing the error information in the same row doesn't require any independent touching of the catalogs. Likewise, the catalog must be updated to re-enable the subscription, so clearing the error from that same row doesn't require any independent touching of the catalogs.
>
> The error information does not *need* to be included in the catalog, but placing the information in any location that won't survive server restart leaves the user no information about why the subscription got disabled after a restart (or crash + restart) happens.
>
> Furthermore, since v2 removed the "disabled_by_error" field in favor of just using subenabled + suberrmsg to determine if the subscription was automatically disabled, not having the information in the catalog would make it ambiguous whether the subscription was manually or automatically disabled.
Is it really useful to write only error message to the system catalog?
Even if we see the error message like "duplicate key value violates
unique constraint “test_tab_pkey”” on the system catalog, we will end
up needing to check the server log for details to properly resolve the
conflict. If the user wants to know whether the subscription is
disabled manually or automatically, the error message on the system
catalog might not necessarily be necessary.
> For the problem in [1], having the xid is more important than it is in my patch, because the user is expected in [1] to use the xid as a handle. But that seems like an odd interface to me. Imagine that a transaction on the publisher side inserted a batch of data, and only a subset of that data conflicts on the subscriber side. What advantage is there in skipping the entire transaction? Wouldn't the user rather skip just the problematic rows? I understand that on the subscriber side it is difficult to do so, but if you are going to implement this sort of thing, it makes more sense to allow the user to filter out data that is problematic rather than filtering out xids that are problematic, and the filter shouldn't just be an in-or-out filter, but rather a mapping function that can redirect the data someplace else or rewrite it before inserting or change the pre-existing conflicting data prior to applying the problematic data or whatever. That's a huge effort, of course, but if the idea in [1] goes in that direction, I don't want my patch to have already added an xid field which ultimately nobody wants.
>
The feature discussed in that thread is meant to be a repair tool for
the subscription in emergency cases when something that should not
have happened happened. I guess that resolving row (or column) level
conflict should be done in another way, for example, by defining
policies for each type of conflict.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-21 02:26 Mark Dilger <[email protected]>
parent: Masahiko Sawada <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: Mark Dilger @ 2021-06-21 02:26 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Jun 20, 2021, at 7:17 PM, Masahiko Sawada <[email protected]> wrote:
>
> I will submit the patch.
Great, thanks!
> There was a discussion that the skipping transaction patch would also
> need to have a feature that tells users the details of the last
> failure transaction such as its XID, timestamp, action etc. In that
> sense, those two patches might need the common infrastructure that the
> apply workers leave the error details somewhere so that the users can
> see it.
Right. Subscription on error triggers would need that, too, if we wrote them.
> Is it really useful to write only error message to the system catalog?
> Even if we see the error message like "duplicate key value violates
> unique constraint “test_tab_pkey”” on the system catalog, we will end
> up needing to check the server log for details to properly resolve the
> conflict. If the user wants to know whether the subscription is
> disabled manually or automatically, the error message on the system
> catalog might not necessarily be necessary.
We can put more information in there. I don't feel strongly about it. I'll wait for your patch to see what infrastructure you need.
> The feature discussed in that thread is meant to be a repair tool for
> the subscription in emergency cases when something that should not
> have happened happened. I guess that resolving row (or column) level
> conflict should be done in another way, for example, by defining
> policies for each type of conflict.
I understand that is the idea, but I'm having trouble believing it will work that way in practice. If somebody has a subscription that has gone awry, what reason do we have to believe there will only be one transaction that will need to be manually purged? It seems just as likely that there would be a million transactions that need to be purged, and creating an interface for users to manually review them and keep or discard on a case by case basis seems unworkable. Sure, you might have specific cases where the number of transactions to purge is small, but I don't like designing the feature around that assumption.
All the same, I'm looking forward to seeing your patch!
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-21 02:53 Amit Kapila <[email protected]>
parent: Mark Dilger <[email protected]>
2 siblings, 0 replies; 74+ messages in thread
From: Amit Kapila @ 2021-06-21 02:53 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Sawada Masahiko <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sat, Jun 19, 2021 at 8:14 PM Mark Dilger
<[email protected]> wrote:
>
> > On Jun 19, 2021, at 3:17 AM, Amit Kapila <[email protected]> wrote:
> > Also, why not also log the xid of the failed
> > transaction?
>
> We could also do that. Reading [1], it seems you are overly focused on user-facing xids. The errdetail in the examples I've been using for testing, and the one mentioned in [1], contain information about the conflicting data. I think users are more likely to understand that a particular primary key value cannot be replicated because it is not unique than to understand that a particular xid cannot be replicated. (Likewise for permissions errors.) For example:
>
> 2021-06-18 16:25:20.139 PDT [56926] ERROR: duplicate key value violates unique constraint "s1_tbl_unique"
> 2021-06-18 16:25:20.139 PDT [56926] DETAIL: Key (i)=(1) already exists.
> 2021-06-18 16:25:20.139 PDT [56926] CONTEXT: COPY tbl, line 2
>
> This tells the user what they need to clean up before they can continue. Telling them which xid tried to apply the change, but not the change itself or the conflict itself, seems rather unhelpful. So at best, the xid is an additional piece of information. I'd rather have both the ERROR and DETAIL fields above and not the xid than have the xid and lack one of those two fields. Even so, I have not yet included the DETAIL field because I didn't want to bloat the catalog.
>
I never said that we don't need the error information. I think we need
xid along with other things.
> For the problem in [1], having the xid is more important than it is in my patch, because the user is expected in [1] to use the xid as a handle. But that seems like an odd interface to me. Imagine that a transaction on the publisher side inserted a batch of data, and only a subset of that data conflicts on the subscriber side. What advantage is there in skipping the entire transaction? Wouldn't the user rather skip just the problematic rows?
>
I think skipping some changes but not others can make the final
transaction data inconsistent. Say, we have a case where, in a
transaction after insert, there is an update or delete on the same
row, then we might silently skip such updates/deletes unless the same
row is already present in the subscriber. I think skipping the entire
transaction based on user instruction would be safer than skipping
some changes that lead to an error.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-21 03:09 Amit Kapila <[email protected]>
parent: Mark Dilger <[email protected]>
1 sibling, 3 replies; 74+ messages in thread
From: Amit Kapila @ 2021-06-21 03:09 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 21, 2021 at 7:56 AM Mark Dilger
<[email protected]> wrote:
>
> > On Jun 20, 2021, at 7:17 PM, Masahiko Sawada <[email protected]> wrote:
> >
> > I will submit the patch.
>
> Great, thanks!
>
> > There was a discussion that the skipping transaction patch would also
> > need to have a feature that tells users the details of the last
> > failure transaction such as its XID, timestamp, action etc. In that
> > sense, those two patches might need the common infrastructure that the
> > apply workers leave the error details somewhere so that the users can
> > see it.
>
> Right. Subscription on error triggers would need that, too, if we wrote them.
>
> > Is it really useful to write only error message to the system catalog?
> > Even if we see the error message like "duplicate key value violates
> > unique constraint “test_tab_pkey”” on the system catalog, we will end
> > up needing to check the server log for details to properly resolve the
> > conflict. If the user wants to know whether the subscription is
> > disabled manually or automatically, the error message on the system
> > catalog might not necessarily be necessary.
> >
I think the two key points are (a) to define exactly what all
information is required to be logged on error, (b) where do we want to
store the information based on requirements. I see that for (b) Mark
is inclined to use the existing catalog table. I feel that is worth
considering but not sure if that is the best way to deal with it. For
example, if we store that information in the catalog, we might need to
consider storing it both in pg_subscription and pg_subscription_rel,
otherwise, we might overwrite the errors as I think what is happening
in the currently proposed patch. The other possibilities could be to
define a new catalog table to capture the error information or log the
required information via stats collector and then the user can see
that info via some stats view.
>
> We can put more information in there. I don't feel strongly about it. I'll wait for your patch to see what infrastructure you need.
>
> > The feature discussed in that thread is meant to be a repair tool for
> > the subscription in emergency cases when something that should not
> > have happened happened. I guess that resolving row (or column) level
> > conflict should be done in another way, for example, by defining
> > policies for each type of conflict.
>
> I understand that is the idea, but I'm having trouble believing it will work that way in practice. If somebody has a subscription that has gone awry, what reason do we have to believe there will only be one transaction that will need to be manually purged?
>
Because currently, we don't proceed after an error unless it is
resolved. Why do you think there could be multiple such transactions?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-21 03:50 Masahiko Sawada <[email protected]>
parent: Amit Kapila <[email protected]>
2 siblings, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2021-06-21 03:50 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 21, 2021 at 12:09 PM Amit Kapila <[email protected]> wrote:
>
> On Mon, Jun 21, 2021 at 7:56 AM Mark Dilger
> <[email protected]> wrote:
> >
> > > On Jun 20, 2021, at 7:17 PM, Masahiko Sawada <[email protected]> wrote:
> > >
> > > I will submit the patch.
> >
> > Great, thanks!
> >
> > > There was a discussion that the skipping transaction patch would also
> > > need to have a feature that tells users the details of the last
> > > failure transaction such as its XID, timestamp, action etc. In that
> > > sense, those two patches might need the common infrastructure that the
> > > apply workers leave the error details somewhere so that the users can
> > > see it.
> >
> > Right. Subscription on error triggers would need that, too, if we wrote them.
> >
> > > Is it really useful to write only error message to the system catalog?
> > > Even if we see the error message like "duplicate key value violates
> > > unique constraint “test_tab_pkey”” on the system catalog, we will end
> > > up needing to check the server log for details to properly resolve the
> > > conflict. If the user wants to know whether the subscription is
> > > disabled manually or automatically, the error message on the system
> > > catalog might not necessarily be necessary.
> > >
>
> I think the two key points are (a) to define exactly what all
> information is required to be logged on error,
When it comes to the patch for skipping transactions, it would
somewhat depend on how users specify transactions to skip. On the
other hand, for this patch, the minimal information would be whether
the subscription is disabled automatically by the server.
> (b) where do we want to
> store the information based on requirements. I see that for (b) Mark
> is inclined to use the existing catalog table. I feel that is worth
> considering but not sure if that is the best way to deal with it. For
> example, if we store that information in the catalog, we might need to
> consider storing it both in pg_subscription and pg_subscription_rel,
> otherwise, we might overwrite the errors as I think what is happening
> in the currently proposed patch. The other possibilities could be to
> define a new catalog table to capture the error information or log the
> required information via stats collector and then the user can see
> that info via some stats view.
This point is also related to the point whether or not that
information needs to last after the server crash (and restart). When
it comes to the patch for skipping transactions, there was a
discussion that we don’t necessarily need it since the tools will be
used in rare cases. But for this proposed patch, I guess it would be
useful if it does. It might be worth considering doing a different way
for each patch. For example, we send the details of last failure
transaction to the stats collector while updating subenabled to
something like “automatically-disabled” instead of to just “false” (or
using another column to show the subscriber is disabled automatically
by the server).
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-21 04:54 Mark Dilger <[email protected]>
parent: Amit Kapila <[email protected]>
2 siblings, 1 reply; 74+ messages in thread
From: Mark Dilger @ 2021-06-21 04:54 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Jun 20, 2021, at 8:09 PM, Amit Kapila <[email protected]> wrote:
>
> Because currently, we don't proceed after an error unless it is
> resolved. Why do you think there could be multiple such transactions?
Just as one example, if the subscriber has a unique index that the publisher lacks, any number of transactions could add non-unique data that then fails to apply on the subscriber. My patch took the view that the user should figure out how to get the subscriber side consistent with the publisher side, but if you instead take the approach that problematic commits should be skipped, it would seem that arbitrarily many such transactions could be committed on the publisher side.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-21 05:11 Amit Kapila <[email protected]>
parent: Mark Dilger <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: Amit Kapila @ 2021-06-21 05:11 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 21, 2021 at 10:24 AM Mark Dilger
<[email protected]> wrote:
>
> > On Jun 20, 2021, at 8:09 PM, Amit Kapila <[email protected]> wrote:
> >
> > Because currently, we don't proceed after an error unless it is
> > resolved. Why do you think there could be multiple such transactions?
>
> Just as one example, if the subscriber has a unique index that the publisher lacks, any number of transactions could add non-unique data that then fails to apply on the subscriber.
>
Then also it will fail on the first such conflict, so even without
your patch, the apply worker corresponding to the subscription won't
be able to proceed after the first error, it won't lead to multiple
failing xids. However, I see a different case where there could be
multiple failing xids and that can happen during initial table sync
where multiple workers failed due to some error. I am not sure your
patch would be able to capture all such failed transactions because
you are recording this information in pg_subscription and not in
pg_subscription_rel.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-21 05:12 Mark Dilger <[email protected]>
parent: Amit Kapila <[email protected]>
2 siblings, 0 replies; 74+ messages in thread
From: Mark Dilger @ 2021-06-21 05:12 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Jun 20, 2021, at 8:09 PM, Amit Kapila <[email protected]> wrote:
>
> (a) to define exactly what all
> information is required to be logged on error, (b) where do we want to
> store the information based on requirements.
I'm not sure it has to be stored anywhere durable. I have a patch in the works to do something like:
create function foreign_key_insert_violation_before() returns conflict_trigger as $$
BEGIN
RAISE NOTICE 'elevel: %', TG_ELEVEL:
RAISE NOTICE 'sqlerrcode: %', TG_SQLERRCODE:
RAISE NOTICE 'message: %', TG_MESSAGE:
RAISE NOTICE 'detail: %', TG_DETAIL:
RAISE NOTICE 'detail_log: %', TG_DETAIL_LOG:
RAISE NOTICE 'hint: %', TG_HINT:
RAISE NOTICE 'schema: %', TG_SCHEMA_NAME:
RAISE NOTICE 'table: %', TG_TABLE_NAME:
RAISE NOTICE 'column: %', TG_COLUMN_NAME:
RAISE NOTICE 'datatype: %', TG_DATATYPE_NAME:
RAISE NOTICE 'constraint: %', TG_CONSTRAINT_NAME:
-- do something useful to prepare for retry of transaction
-- which raised a foreign key violation
END
$$ language plpgsql;
create function foreign_key_insert_violation_after() returns conflict_trigger as $$
BEGIN
-- do something useful to cleanup after retry of transaction
-- which raised a foreign key violation
END
$$ language plpgsql;
create conflict trigger regress_conflict_trigger_insert on regress_conflictsub
before foreign_key_violation
when tag in ('INSERT')
execute procedure foreign_key_insert_violation_before();
create conflict trigger regress_conflict_trigger_insert on regress_conflictsub
after foreign_key_violation
when tag in ('INSERT')
execute procedure foreign_key_insert_violation_after();
The idea is that, for subscriptions that have conflict triggers defined, the apply will be wrapped in a PG_TRY()/PG_CATCH() block. If it fails, the ErrorData will be copied in the ConflictTriggerContext, and then the transaction will be attempted again, but this time with any BEFORE and AFTER triggers applied. The triggers could then return a special result indicating whether the transaction should be permanently skipped, applied, or whatever. None of the data needs to be stored anywhere non-transient, as it just gets handed to the triggers.
I think the other patch is a subset of this functionality, as using this system to create triggers which query a table containing transactions to be skipped would be enough to get the functionality you've been discussing. But this system could also do other things, like modify data. Admittedly, this is akin to a statement level trigger and not a row level trigger, so a number of things you might want to do would be hard to do from this. But perhaps the equivalent of row level triggers could also be written?
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-21 05:16 Mark Dilger <[email protected]>
parent: Amit Kapila <[email protected]>
1 sibling, 0 replies; 74+ messages in thread
From: Mark Dilger @ 2021-06-21 05:16 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Jun 20, 2021, at 10:11 PM, Amit Kapila <[email protected]> wrote:
>
> Then also it will fail on the first such conflict, so even without
> your patch, the apply worker corresponding to the subscription won't
> be able to proceed after the first error, it won't lead to multiple
> failing xids.
I'm not sure we're talking about the same thing. I'm saying that if the user is expected to clear each error manually, there could be many such errors for them to clear. It may be true that the second error doesn't occur on the subscriber side until after the first is cleared, but that still leaves the user having to clear one after the next until arbitrarily many of them coming from the publisher side are cleared.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-21 05:25 Mark Dilger <[email protected]>
parent: Amit Kapila <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Mark Dilger @ 2021-06-21 05:25 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Jun 20, 2021, at 10:11 PM, Amit Kapila <[email protected]> wrote:
>
> However, I see a different case where there could be
> multiple failing xids and that can happen during initial table sync
> where multiple workers failed due to some error. I am not sure your
> patch would be able to capture all such failed transactions because
> you are recording this information in pg_subscription and not in
> pg_subscription_rel.
Right, I wasn't trying to capture everything, just enough to give the user a reasonable indication of what went wrong. My patch was designed around the idea that the user would need to figure out how to fix the subscriber side prior to re-enabling the subscription. As such, I wasn't bothered with trying to store everything, just enough to give the user a clue where to look. I don't mind if you want to store more information, and maybe that needs to be stored somewhere else. Do you believe pg_subscription_rel is a suitable location?
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-21 05:26 Amit Kapila <[email protected]>
parent: Masahiko Sawada <[email protected]>
0 siblings, 0 replies; 74+ messages in thread
From: Amit Kapila @ 2021-06-21 05:26 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 21, 2021 at 9:21 AM Masahiko Sawada <[email protected]> wrote:
>
> On Mon, Jun 21, 2021 at 12:09 PM Amit Kapila <[email protected]> wrote:
> >
> > On Mon, Jun 21, 2021 at 7:56 AM Mark Dilger
> > <[email protected]> wrote:
> > >
> > > > On Jun 20, 2021, at 7:17 PM, Masahiko Sawada <[email protected]> wrote:
> > > >
> > > > I will submit the patch.
> > >
> > > Great, thanks!
> > >
> > > > There was a discussion that the skipping transaction patch would also
> > > > need to have a feature that tells users the details of the last
> > > > failure transaction such as its XID, timestamp, action etc. In that
> > > > sense, those two patches might need the common infrastructure that the
> > > > apply workers leave the error details somewhere so that the users can
> > > > see it.
> > >
> > > Right. Subscription on error triggers would need that, too, if we wrote them.
> > >
> > > > Is it really useful to write only error message to the system catalog?
> > > > Even if we see the error message like "duplicate key value violates
> > > > unique constraint “test_tab_pkey”” on the system catalog, we will end
> > > > up needing to check the server log for details to properly resolve the
> > > > conflict. If the user wants to know whether the subscription is
> > > > disabled manually or automatically, the error message on the system
> > > > catalog might not necessarily be necessary.
> > > >
> >
> > I think the two key points are (a) to define exactly what all
> > information is required to be logged on error,
>
> When it comes to the patch for skipping transactions, it would
> somewhat depend on how users specify transactions to skip. On the
> other hand, for this patch, the minimal information would be whether
> the subscription is disabled automatically by the server.
>
True, but still there will be some information related to ERROR which
we wanted the user to see unless we ask them to refer to logs for
that.
> > (b) where do we want to
> > store the information based on requirements. I see that for (b) Mark
> > is inclined to use the existing catalog table. I feel that is worth
> > considering but not sure if that is the best way to deal with it. For
> > example, if we store that information in the catalog, we might need to
> > consider storing it both in pg_subscription and pg_subscription_rel,
> > otherwise, we might overwrite the errors as I think what is happening
> > in the currently proposed patch. The other possibilities could be to
> > define a new catalog table to capture the error information or log the
> > required information via stats collector and then the user can see
> > that info via some stats view.
>
> This point is also related to the point whether or not that
> information needs to last after the server crash (and restart). When
> it comes to the patch for skipping transactions, there was a
> discussion that we don’t necessarily need it since the tools will be
> used in rare cases. But for this proposed patch, I guess it would be
> useful if it does. It might be worth considering doing a different way
> for each patch. For example, we send the details of last failure
> transaction to the stats collector while updating subenabled to
> something like “automatically-disabled” instead of to just “false” (or
> using another column to show the subscriber is disabled automatically
> by the server).
>
I agree that it is worth considering to have subenabled to have a
tri-state (enable, disabled, automatically-disabled) value instead of
just a boolean. But in this case, if the stats collector missed
updating the information, the user may have to manually update the
subscription and let the error happen again to see it.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-21 05:49 Amit Kapila <[email protected]>
parent: Mark Dilger <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Amit Kapila @ 2021-06-21 05:49 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 21, 2021 at 10:55 AM Mark Dilger
<[email protected]> wrote:
>
> > On Jun 20, 2021, at 10:11 PM, Amit Kapila <[email protected]> wrote:
> >
> > However, I see a different case where there could be
> > multiple failing xids and that can happen during initial table sync
> > where multiple workers failed due to some error. I am not sure your
> > patch would be able to capture all such failed transactions because
> > you are recording this information in pg_subscription and not in
> > pg_subscription_rel.
>
> Right, I wasn't trying to capture everything, just enough to give the user a reasonable indication of what went wrong. My patch was designed around the idea that the user would need to figure out how to fix the subscriber side prior to re-enabling the subscription. As such, I wasn't bothered with trying to store everything, just enough to give the user a clue where to look.
>
Okay, but the clue will be pretty random because you might end up just
logging one out of several errors.
> I don't mind if you want to store more information, and maybe that needs to be stored somewhere else. Do you believe pg_subscription_rel is a suitable location?
>
It won't be sufficient to store information in either
pg_subscription_rel or pg_susbscription. I think if we want to store
the required information in a catalog then we need to define a new
catalog (pg_subscription_conflicts or something like that) with
information corresponding to each rel in subscription (srsubid oid
(Reference to subscription), srrelid oid (Reference to relation),
<columns for error_info>). OTOH, we can choose to send the error
information to stats collector which will then be available via stat
view and update system catalog to disable the subscription but there
will be a risk that we might send info of failed transaction to stats
collector but then fail to update system catalog to disable the
subscription.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-21 10:47 Amit Kapila <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 3 replies; 74+ messages in thread
From: Amit Kapila @ 2021-06-21 10:47 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 21, 2021 at 11:19 AM Amit Kapila <[email protected]> wrote:
>
> On Mon, Jun 21, 2021 at 10:55 AM Mark Dilger
> <[email protected]> wrote:
>
> > I don't mind if you want to store more information, and maybe that needs to be stored somewhere else. Do you believe pg_subscription_rel is a suitable location?
> >
> It won't be sufficient to store information in either
> pg_subscription_rel or pg_susbscription. I think if we want to store
> the required information in a catalog then we need to define a new
> catalog (pg_subscription_conflicts or something like that) with
> information corresponding to each rel in subscription (srsubid oid
> (Reference to subscription), srrelid oid (Reference to relation),
> <columns for error_info>). OTOH, we can choose to send the error
> information to stats collector which will then be available via stat
> view and update system catalog to disable the subscription but there
> will be a risk that we might send info of failed transaction to stats
> collector but then fail to update system catalog to disable the
> subscription.
>
I think we should store the input from the user (like disable_on_error
flag or xid to skip) in the system catalog pg_subscription and send
the error information (subscrtion_id, rel_id, xid of failed xact,
error_code, error_message, etc.) to the stats collector which can be
used to display such information via a stat view.
The disable_on_error flag handling could be that on error it sends the
required error info to stats collector and then updates the subenabled
in pg_subscription. In rare conditions, where we are able to send the
message but couldn't update the subenabled info in pg_subscription
either due to some error or server restart, the apply worker would
again try to apply the same change and would hit the same error again
which I think should be fine because it will ultimately succeed.
The skip xid handling will also be somewhat similar where on an error,
we will send the error information to stats collector which will be
displayed via stats view. Then the user is expected to ask for skip
xid (Alter Subscription ... SKIP <xid_value>) based on information
displayed via stat view. Now, the apply worker can skip changes from
such a transaction, and then during processing of commit record of the
skipped transaction, it should update xid to invalid value, so that
next time that shouldn't be used. I think it is important to update
xid to an invalid value as part of the skipped transaction because
otherwise, after the restart, we won't be able to decide whether we
still want to skip the xid stored for a subscription.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-22 00:57 Peter Smith <[email protected]>
parent: Amit Kapila <[email protected]>
2 siblings, 4 replies; 74+ messages in thread
From: Peter Smith @ 2021-06-22 00:57 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Mark Dilger <[email protected]>; Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
Much of the discussion above seems to be related to where to store the
error information and how much information is needed to be useful.
As a summary, the 5 alternatives I have seen mentioned are:
#1. Store some simple message in the pg_subscription ("I wasn't trying
to capture everything, just enough to give the user a reasonable
indication of what went wrong" [Mark-1]). Storing the error message
was also seen as a convenience for writing TAP tests ("I originally
ran into the motivation to write this patch when frustrated that TAP
tests needed to parse the apply worker log file" [Mark-2}). It also
can sometimes provide a simple clue for the error (e.g. PK violation
for table TBL) but still the user will have to look elsewhere for
details to resolve the error. So while this implementation seems good
for simple scenarios, it appears to have been shot down because the
non-trivial scenarios either have insufficient or wrong information in
the error message. Some DETAILS could have been added to give more
information but that would maybe bloat the catalog ("I have not yet
included the DETAIL field because I didn't want to bloat the catalog."
[Mark-3])
#2. Similarly another idea was to use another existing catalog table
pg_subscription_rel. This could have the same problems ("It won't be
sufficient to store information in either pg_subscription_rel or
pg_susbscription." [Amit-1])
#3. There is another suggestion to use the Stats Collector to hold the
error message [Amit-2]. For me, this felt like blurring too much the
distinction between "stats tracking/metrics" and "logs". ERROR logs
must be flushed, whereas for stats (IIUC) there is no guarantee that
everything you need to see would be present. Indeed Amit wrote "But in
this case, if the stats collector missed updating the information, the
user may have to manually update the subscription and let the error
happen again to see it." [Amit-3]. Requesting the user to cause the
same error again just in case it was not captured a first time seems
too strange to me.
#4. The next idea was to have an entirely new catalog for holding the
subscription error information. I feel that storing/duplicating lots
of error information in another table seems like a bridge too far.
What about the risks of storing incorrect or sufficient information?
What is the advantage of duplicating errors over just referring to the
log files for ERROR details?
#5. Document to refer to the logs. All ERROR details are already in
the logs, and this seems to me the intuitive place to look for them.
Searching for specific errors becomes difficult programmatically (is
this really a problem other than complex TAP tests?). But here there
is no risk of missing or insufficient information captured in the log
files ("but still there will be some information related to ERROR
which we wanted the user to see unless we ask them to refer to logs
for that." [Amit-4}).
---
My preferred alternative is #5. ERRORs are logged in the log file, so
there is nothing really for this patch to do in this regard (except
documentation), and there is no risk of missing any information, no
ambiguity of having duplicated errors, and it is the intuitive place
the user would look.
So I felt current best combination is just this:
a) A tri-state indicating the state of the subscription: e.g.
something like "enabled" ('e')/ "disabled" ('d') / "auto-disabled"
('a') [Amit-5]
b) For "auto-disabled" the PG docs would be updated tell the user to
check the logs to resolve the problem before re-enabling the
subscription
//////////
IMO it is not made exactly clear to me what is the main goal of this
patch. Because of this, I feel that you can't really judge if this new
option is actually useful or not except only in hindsight. It seems
like whatever you implement can be made to look good or bad, just by
citing different test scenarios.
e.g.
* Is the goal mainly to help automated (TAP) testing? In that case,
then maybe you do want to store the error message somewhere other than
the log files. But still I wonder if results would be unpredictable
anyway - e.g if there are multiple tables all with errors then it
depends on the tablesync order of execution which error you see caused
the auto-disable, right? And if it is not predictable maybe it is less
useful.
* Is the goal to prevent some *unattended* SUBSCRIPTION from going bad
at some point in future and then going into a relaunch loop for
days/weeks and causing 1000's of errors without the user noticing. In
that case, this patch seems to be quite useful, but for this goal
maybe you don't want to be checking the tablesync workers at all, but
should only be checking the apply worker like your original v1 patch
did.
* Is the goal just to be a convenient way to disable the subscription
during the CREATE SUBSCRIPTION phase so that the user can make
corrections in peace without the workers re-launching and making more
error logs? Here the patch is helpful, but only for simple scenarios
like 1 faulty table. Imagine if there are 10 tables (all with PK
violations at DATASYNC copy) then you will encounter them one at a
time and have to re-enable the subscription 10 times, after fixing
each error in turn. So in this scenario the new option might be more
of a hindrance than a help because it would be easier if the user just
did "ALTER SUBSCRIPTION sub DISABLE" manually and fixed all the
problems in one sitting before re-enabling.
* etc
//////////
Finally, here is one last (crazy?) thought-bubble just for
consideration. I might be wrong, but my gut feeling is that the Stats
Collector is intended more for "tracking" and for "metrics" rather
than for holding duplicates of logged error messages. At the same
time, I felt that disabling an entire subscription due to a single
rogue error might be overkill sometimes. But I wonder if there is a
way to combine those two ideas so that the Stats Collector gets some
new counter for tracking the number of worker re-launches that have
occurred, meanwhile there could be a subscription option which gives a
threshold above which you would disable the subscription.
e.g.
"disable_on_error_threshold=0" default, relaunch forever
"disable_on_error_threshold=1" disable upon first error encountered.
(This is how your patch behaves now I think.)
"disable_on_error_threshold=500" disable if the re-launch errors go
unattended and happen 500 times.
------
[Mark-1] https://www.postgresql.org/message-id/A539C848-670E-454F-B31C-82D3CBE9F5AC%40enterprisedb.com
[Mark-2] https://www.postgresql.org/message-id/DB35438F-9356-4841-89A0-412709EBD3AB%40enterprisedb.com
[Mark-3] https://www.postgresql.org/message-id/DE7E13B7-DC76-416A-A98F-3BC3F80E6BE9%40enterprisedb.com
[Amit-1] https://www.postgresql.org/message-id/CAA4eK1K_JFSFrAkr_fgp3VX6hTSmjK%3DwNs4Tw8rUWHGp0%2BBsaw%40mail...
[Amit-2] https://www.postgresql.org/message-id/CAA4eK1%2BNoRbYSH1J08zi4OJ_EUMcjmxTwnmwVqZ6e_xzS0D6VA%40mail.g...
[Amit-3] https://www.postgresql.org/message-id/CAA4eK1Kyx6U9yxC7OXoBD7pHC3bJ4LuNGd%3DOiABmiW6%2BqG%2BvEQ%40ma...
[Amit-4] https://www.postgresql.org/message-id/CAA4eK1Kyx6U9yxC7OXoBD7pHC3bJ4LuNGd%3DOiABmiW6%2BqG%2BvEQ%40ma...
[Amit-5] https://www.postgresql.org/message-id/CAA4eK1Kyx6U9yxC7OXoBD7pHC3bJ4LuNGd%3DOiABmiW6%2BqG%2BvEQ%40ma...
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-22 02:29 Mark Dilger <[email protected]>
parent: Peter Smith <[email protected]>
3 siblings, 0 replies; 74+ messages in thread
From: Mark Dilger @ 2021-06-22 02:29 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Jun 21, 2021, at 5:57 PM, Peter Smith <[email protected]> wrote:
>
> #5. Document to refer to the logs. All ERROR details are already in
> the logs, and this seems to me the intuitive place to look for them.
My original motivation came from writing TAP tests to check that the permissions systems would properly deny the apply worker when running under a non-superuser role. The idea is that the user with the responsibility for managing subscriptions won't have enough privilege to read the logs. Whatever information that user needs (if any) must be someplace else.
> Searching for specific errors becomes difficult programmatically (is
> this really a problem other than complex TAP tests?).
I believe there is a problem, because I remain skeptical that these errors will be both existent and rare. Either you've configured your system correctly and you get zero of these, or you've misconfigured it and you get some non-zero number of them. I don't see any reason to assume that number will be small.
The best way to deal with that is to be able to tell the system what to do with them, like "if the error has this error code and the error message matches this regular expression, then do this, else do that." That's why I think allowing triggers to be created on subscriptions makes the most sense (though is probably the hardest system being proposed so far.)
> But here there
> is no risk of missing or insufficient information captured in the log
> files ("but still there will be some information related to ERROR
> which we wanted the user to see unless we ask them to refer to logs
> for that." [Amit-4}).
Not only is there a problem if the user doesn't have permission to view the logs, but also, if we automatically disable the subscription until the error is manually cleared, the logs might be rotated out of existence before the user takes any action. In that case, the logs will be entirely missing, and not even the error message will remain. At least with the patch I submitted, the error message will remain, though I take Amit's point that there are deficiencies in handling parallel tablesync workers, etc.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-22 02:35 Mark Dilger <[email protected]>
parent: Peter Smith <[email protected]>
3 siblings, 0 replies; 74+ messages in thread
From: Mark Dilger @ 2021-06-22 02:35 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Jun 21, 2021, at 5:57 PM, Peter Smith <[email protected]> wrote:
>
> * Is the goal mainly to help automated (TAP) testing?
Absolutely, that was my original motivation. But I don't think that is the primary reason the patch would be accepted. There is a cost to having the logical replication workers attempt ad infinitum to apply a transaction that will never apply.
Also, if you are waiting for a subscription to catch up, it is far from obvious that you will wait forever.
> In that case,
> then maybe you do want to store the error message somewhere other than
> the log files. But still I wonder if results would be unpredictable
> anyway - e.g if there are multiple tables all with errors then it
> depends on the tablesync order of execution which error you see caused
> the auto-disable, right? And if it is not predictable maybe it is less
> useful.
But if you are writing a TAP test, you should be the one controlling whether that is the case. I don't think it would be unpredictable from the point of view of the test author.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-22 02:42 Masahiko Sawada <[email protected]>
parent: Amit Kapila <[email protected]>
2 siblings, 0 replies; 74+ messages in thread
From: Masahiko Sawada @ 2021-06-22 02:42 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 21, 2021 at 7:48 PM Amit Kapila <[email protected]> wrote:
>
> On Mon, Jun 21, 2021 at 11:19 AM Amit Kapila <[email protected]> wrote:
> >
> > On Mon, Jun 21, 2021 at 10:55 AM Mark Dilger
> > <[email protected]> wrote:
> >
> > > I don't mind if you want to store more information, and maybe that needs to be stored somewhere else. Do you believe pg_subscription_rel is a suitable location?
> > >
> > It won't be sufficient to store information in either
> > pg_subscription_rel or pg_susbscription. I think if we want to store
> > the required information in a catalog then we need to define a new
> > catalog (pg_subscription_conflicts or something like that) with
> > information corresponding to each rel in subscription (srsubid oid
> > (Reference to subscription), srrelid oid (Reference to relation),
> > <columns for error_info>). OTOH, we can choose to send the error
> > information to stats collector which will then be available via stat
> > view and update system catalog to disable the subscription but there
> > will be a risk that we might send info of failed transaction to stats
> > collector but then fail to update system catalog to disable the
> > subscription.
> >
>
> I think we should store the input from the user (like disable_on_error
> flag or xid to skip) in the system catalog pg_subscription and send
> the error information (subscrtion_id, rel_id, xid of failed xact,
> error_code, error_message, etc.) to the stats collector which can be
> used to display such information via a stat view.
>
> The disable_on_error flag handling could be that on error it sends the
> required error info to stats collector and then updates the subenabled
> in pg_subscription. In rare conditions, where we are able to send the
> message but couldn't update the subenabled info in pg_subscription
> either due to some error or server restart, the apply worker would
> again try to apply the same change and would hit the same error again
> which I think should be fine because it will ultimately succeed.
>
> The skip xid handling will also be somewhat similar where on an error,
> we will send the error information to stats collector which will be
> displayed via stats view. Then the user is expected to ask for skip
> xid (Alter Subscription ... SKIP <xid_value>) based on information
> displayed via stat view. Now, the apply worker can skip changes from
> such a transaction, and then during processing of commit record of the
> skipped transaction, it should update xid to invalid value, so that
> next time that shouldn't be used. I think it is important to update
> xid to an invalid value as part of the skipped transaction because
> otherwise, after the restart, we won't be able to decide whether we
> still want to skip the xid stored for a subscription.
Sounds reasonable.
The feature that sends the error information to the stats collector is
a common feature for both and itself is also useful. As discussed in
that skip transaction patch thread, it would also be good if we write
error information (relation, action, xid, etc) to the server log too.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-22 02:44 Amit Kapila <[email protected]>
parent: Peter Smith <[email protected]>
3 siblings, 0 replies; 74+ messages in thread
From: Amit Kapila @ 2021-06-22 02:44 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Mark Dilger <[email protected]>; Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Jun 22, 2021 at 6:27 AM Peter Smith <[email protected]> wrote:
>
> #3. There is another suggestion to use the Stats Collector to hold the
> error message [Amit-2]. For me, this felt like blurring too much the
> distinction between "stats tracking/metrics" and "logs". ERROR logs
> must be flushed, whereas for stats (IIUC) there is no guarantee that
> everything you need to see would be present. Indeed Amit wrote "But in
> this case, if the stats collector missed updating the information, the
> user may have to manually update the subscription and let the error
> happen again to see it." [Amit-3]. Requesting the user to cause the
> same error again just in case it was not captured a first time seems
> too strange to me.
>
I don't think it will often be the case that the stats collector will
miss updating the information. I am not feeling comfortable storing
error information in system catalogs. We have some other views which
capture somewhat similar conflict information
(pg_stat_database_conflicts) or failed transactions information. So, I
thought here we are extending the similar concept by storing some
additional information about errors.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-22 02:49 Mark Dilger <[email protected]>
parent: Peter Smith <[email protected]>
3 siblings, 0 replies; 74+ messages in thread
From: Mark Dilger @ 2021-06-22 02:49 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Jun 21, 2021, at 5:57 PM, Peter Smith <[email protected]> wrote:
>
> * Is the goal to prevent some *unattended* SUBSCRIPTION from going bad
> at some point in future and then going into a relaunch loop for
> days/weeks and causing 1000's of errors without the user noticing. In
> that case, this patch seems to be quite useful, but for this goal
> maybe you don't want to be checking the tablesync workers at all, but
> should only be checking the apply worker like your original v1 patch
> did.
Yeah, my motivation was preventing an infinite loop, and providing a clean way for the users to know that replication they are waiting for won't ever complete, rather than having to infer that it will never halt.
> * Is the goal just to be a convenient way to disable the subscription
> during the CREATE SUBSCRIPTION phase so that the user can make
> corrections in peace without the workers re-launching and making more
> error logs?
No. This is not and never was my motivation. It's an interesting question, but that idea never crossed my mind. I'm not sure what changes somebody would want to make *after* creating the subscription. Certainly, there may be problems with how they have things set up, but they won't know that until the first error happens.
> Here the patch is helpful, but only for simple scenarios
> like 1 faulty table. Imagine if there are 10 tables (all with PK
> violations at DATASYNC copy) then you will encounter them one at a
> time and have to re-enable the subscription 10 times, after fixing
> each error in turn.
You are assuming disable_on_error=true. It is false by default. But ok, let's accept that assumption for the sake of argument. Now, will you have to manually go through the process 10 times? I'm not sure. The user might figure out their mistake after seeing the first error.
> So in this scenario the new option might be more
> of a hindrance than a help because it would be easier if the user just
> did "ALTER SUBSCRIPTION sub DISABLE" manually and fixed all the
> problems in one sitting before re-enabling.
Yeah, but since the new option is off by default, I don't see any sensible complaint.
>
> * etc
>
> //////////
>
> Finally, here is one last (crazy?) thought-bubble just for
> consideration. I might be wrong, but my gut feeling is that the Stats
> Collector is intended more for "tracking" and for "metrics" rather
> than for holding duplicates of logged error messages. At the same
> time, I felt that disabling an entire subscription due to a single
> rogue error might be overkill sometimes.
I'm happy to entertain criticism of the particulars of how my patch approaches this problem, but it is already making a distinction between transient errors (resources, network, etc.) vs. ones that are non-transient. Again, I might not have drawn the line in the right place, but the patch is not intended to disable subscriptions in response to transient errors.
> But I wonder if there is a
> way to combine those two ideas so that the Stats Collector gets some
> new counter for tracking the number of worker re-launches that have
> occurred, meanwhile there could be a subscription option which gives a
> threshold above which you would disable the subscription.
> e.g.
> "disable_on_error_threshold=0" default, relaunch forever
> "disable_on_error_threshold=1" disable upon first error encountered.
> (This is how your patch behaves now I think.)
> "disable_on_error_threshold=500" disable if the re-launch errors go
> unattended and happen 500 times.
That sounds like a misfeature to me. You could have a subscription that works fine for a month, surviving numerous short network outages, but then gets autodisabled after a longer network outage. I'm not sure why anybody would want that. You might argue for exponential backoff, where it never gets autodisabled on transient errors, but retries less frequently. But I don't want to expand the scope of this patch to include that, at least not without a lot more evidence that it is needed.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-23 02:53 Amit Kapila <[email protected]>
parent: Amit Kapila <[email protected]>
2 siblings, 0 replies; 74+ messages in thread
From: Amit Kapila @ 2021-06-23 02:53 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 21, 2021 at 4:17 PM Amit Kapila <[email protected]> wrote:
>
> On Mon, Jun 21, 2021 at 11:19 AM Amit Kapila <[email protected]> wrote:
> >
>
> I think we should store the input from the user (like disable_on_error
> flag or xid to skip) in the system catalog pg_subscription and send
> the error information (subscrtion_id, rel_id, xid of failed xact,
> error_code, error_message, etc.) to the stats collector which can be
> used to display such information via a stat view.
>
> The disable_on_error flag handling could be that on error it sends the
> required error info to stats collector and then updates the subenabled
> in pg_subscription. In rare conditions, where we are able to send the
> message but couldn't update the subenabled info in pg_subscription
> either due to some error or server restart, the apply worker would
> again try to apply the same change and would hit the same error again
> which I think should be fine because it will ultimately succeed.
>
> The skip xid handling will also be somewhat similar where on an error,
> we will send the error information to stats collector which will be
> displayed via stats view. Then the user is expected to ask for skip
> xid (Alter Subscription ... SKIP <xid_value>) based on information
> displayed via stat view. Now, the apply worker can skip changes from
> such a transaction, and then during processing of commit record of the
> skipped transaction, it should update xid to invalid value, so that
> next time that shouldn't be used. I think it is important to update
> xid to an invalid value as part of the skipped transaction because
> otherwise, after the restart, we won't be able to decide whether we
> still want to skip the xid stored for a subscription.
>
One minor detail I missed in the above sketch for skipped transaction
feature was that actually we only need replication origin state from
the commit record of the skipped transaction and then I think we need
to start a transaction, update the xid value to invalid, set the
replication origin state and commit that transaction.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-06-28 04:47 Masahiko Sawada <[email protected]>
parent: Mark Dilger <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2021-06-28 04:47 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 21, 2021 at 11:26 AM Mark Dilger
<[email protected]> wrote:
>
>
>
> > On Jun 20, 2021, at 7:17 PM, Masahiko Sawada <[email protected]> wrote:
> >
> > I will submit the patch.
>
> Great, thanks!
I've submitted the patches on that thread[1]. There are three patches:
skipping the transaction on the subscriber side, reporting error
details in the errcontext, and reporting the error details to the
stats collector. Feedback is very welcome.
[1] https://www.postgresql.org/message-id/CAD21AoBU4jGEO6AXcykQ9y7tat0RrB5--8ZoJgfcj%2BLPs7nFZQ%40mail.g...
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-02 10:42 [email protected] <[email protected]>
parent: Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: [email protected] @ 2021-11-02 10:42 UTC (permalink / raw)
To: 'Masahiko Sawada' <[email protected]>; Mark Dilger <[email protected]>; +Cc: Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Monday, June 28, 2021 1:47 PM Masahiko Sawada <[email protected]> wrote:
> On Mon, Jun 21, 2021 at 11:26 AM Mark Dilger
> <[email protected]> wrote:
> > > On Jun 20, 2021, at 7:17 PM, Masahiko Sawada
> <[email protected]> wrote:
> > >
> > > I will submit the patch.
> >
> > Great, thanks!
>
> I've submitted the patches on that thread[1]. There are three patches:
> skipping the transaction on the subscriber side, reporting error details in the
> errcontext, and reporting the error details to the stats collector. Feedback is
> very welcome.
>
> [1]
> https://www.postgresql.org/message-id/CAD21AoBU4jGEO6AXcykQ9y7tat0R
> rB5--8ZoJgfcj%2BLPs7nFZQ%40mail.gmail.com
Hi, thanks Sawada-san for keep updating the skip xid patch in the thread.
This thread has stopped since the patch submission.
I've rebased the 'disable_on_error' option
so that it can be applied on top of skip xid shared in [1].
I've written Mark Dilger as the original author in the commit message.
This patch is simply rebased to reactive this thread.
So there are still pending item to discuss for example,
how we should deal with multiple errors of several table sync workers.
I extracted only 'disable_on_error' option
because the skip xid and the latest error message fulfill the motivation
to make it easy to write TAP tests already I felt.
[1] - https://www.postgresql.org/message-id/CAD21AoDY-9_x819F_m1_wfCVXXFJrGiSmR2MfC9Nw4nW8Om0qA%40mail.gma...
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v3-Optionally-disabling-subscriptions-on-error.patch (23.5K, ../../TYCPR01MB837384DA6C879A709EBF30D1ED8B9@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v3-Optionally-disabling-subscriptions-on-error.patch)
download | inline diff:
From ae43c41fdfd9025d724b29fc762a6f05bece9a40 Mon Sep 17 00:00:00 2001
From: Osumi Takamichi <[email protected]>
Date: Tue, 2 Nov 2021 09:53:53 +0000
Subject: [PATCH v3] Optionally disabling subscriptions on error
Logical replication apply workers for a subscription can easily get
stuck in an infinite loop of attempting to apply a change,
triggering an error (such as a constraint violation), exiting with
an error written to the subscription worker log, and restarting.
To partially remedy the situation, adding a new
subscription_parameter named 'disable_on_error'. To be consistent
with old behavior, the parameter defaults to false. When true, the
apply worker catches errors thrown, and for errors that are deemed
not to be transient, disables the subscription in order to break the
loop. The error is still also written to the logs.
Proposed and Written originally by Mark Dilger.
---
doc/src/sgml/ref/create_subscription.sgml | 12 ++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 27 +++-
src/backend/replication/logical/launcher.c | 1 +
src/backend/replication/logical/worker.c | 167 +++++++++++++++++++--
src/include/catalog/pg_subscription.h | 4 +
src/test/perl/PostgreSQL/Test/Cluster.pm | 40 +++++
src/test/subscription/t/027_disable_on_error.pl | 192 ++++++++++++++++++++++++
9 files changed, 434 insertions(+), 12 deletions(-)
create mode 100644 src/test/subscription/t/027_disable_on_error.pl
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 990a41f..ef12277 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -142,6 +142,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</varlistentry>
<varlistentry>
+ <term><literal>disable_on_error</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription should be automatically disabled
+ if replicating data from the publisher triggers non-transicent errors
+ such as referential integrity or permissions errors. The default is
+ <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>enabled</literal> (<type>boolean</type>)</term>
<listitem>
<para>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 25021e2..fcace32 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->binary = subform->subbinary;
sub->stream = subform->substream;
sub->twophasestate = subform->subtwophasestate;
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index a2ee00c..8220fca 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1258,7 +1258,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
+GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subdisableonerr, subbinary,
substream, subtwophasestate, subslotname, subsynccommit, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 18962b9..122eed7 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -61,6 +61,7 @@
#define SUBOPT_BINARY 0x00000080
#define SUBOPT_STREAMING 0x00000100
#define SUBOPT_TWOPHASE_COMMIT 0x00000200
+#define SUBOPT_DISABLE_ON_ERR 0x00000400
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -82,6 +83,7 @@ typedef struct SubOpts
bool binary;
bool streaming;
bool twophase;
+ bool disableonerr;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -129,6 +131,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->streaming = false;
if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT))
opts->twophase = false;
+ if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
+ opts->disableonerr = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -248,6 +252,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_TWOPHASE_COMMIT;
opts->twophase = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR) &&
+ strcmp(defel->defname, "disable_on_error") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DISABLE_ON_ERR))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DISABLE_ON_ERR;
+ opts->disableonerr = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -397,7 +410,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
supported_opts = (SUBOPT_CONNECT | SUBOPT_ENABLED | SUBOPT_CREATE_SLOT |
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT);
+ SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
+ SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -465,6 +479,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled);
+ values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary);
values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming);
values[Anum_pg_subscription_subtwophasestate - 1] =
@@ -871,11 +886,19 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
{
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING);
+ SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
+ if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR))
+ {
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = BoolGetDatum(opts.disableonerr);
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_SLOT_NAME))
{
/*
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 3fb4caa..febfc4d 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -132,6 +132,7 @@ get_subscription_list(void)
sub->dbid = subform->subdbid;
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->name = pstrdup(NameStr(subform->subname));
/* We don't fill fields we are not interested in. */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 3a40684..c79fad5 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -136,6 +136,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/partition.h"
#include "catalog/pg_inherits.h"
@@ -334,6 +335,7 @@ static void apply_spooled_messages(TransactionId xid, XLogRecPtr lsn);
static void apply_error_callback(void *arg);
static inline void set_apply_error_context_xact(TransactionId xid, TimestampTz ts);
static inline void reset_apply_error_context_info(void);
+static bool IsSubscriptionDisablingError(void);
/*
* Should this worker apply changes for given relation.
@@ -2752,6 +2754,123 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/*
+ * Errors which are transient, network protocol related, or resource exhaustion
+ * related, should not disable a subscription. These may clear up without user
+ * intervention in the subscription, schema, or data being replicated.
+ */
+static bool
+IsSubscriptionDisablingError(void)
+{
+ switch (geterrcode())
+ {
+ case ERRCODE_CONNECTION_EXCEPTION:
+ case ERRCODE_CONNECTION_DOES_NOT_EXIST:
+ case ERRCODE_CONNECTION_FAILURE:
+ case ERRCODE_SQLCLIENT_UNABLE_TO_ESTABLISH_SQLCONNECTION:
+ case ERRCODE_SQLSERVER_REJECTED_ESTABLISHMENT_OF_SQLCONNECTION:
+ case ERRCODE_TRANSACTION_RESOLUTION_UNKNOWN:
+ case ERRCODE_PROTOCOL_VIOLATION:
+ case ERRCODE_INSUFFICIENT_RESOURCES:
+ case ERRCODE_DISK_FULL:
+ case ERRCODE_OUT_OF_MEMORY:
+ case ERRCODE_TOO_MANY_CONNECTIONS:
+ case ERRCODE_CONFIGURATION_LIMIT_EXCEEDED:
+ case ERRCODE_PROGRAM_LIMIT_EXCEEDED:
+ case ERRCODE_STATEMENT_TOO_COMPLEX:
+ case ERRCODE_TOO_MANY_COLUMNS:
+ case ERRCODE_TOO_MANY_ARGUMENTS:
+ case ERRCODE_OPERATOR_INTERVENTION:
+ case ERRCODE_QUERY_CANCELED:
+ case ERRCODE_ADMIN_SHUTDOWN:
+ case ERRCODE_CRASH_SHUTDOWN:
+ case ERRCODE_CANNOT_CONNECT_NOW:
+ case ERRCODE_DATABASE_DROPPED:
+ case ERRCODE_IDLE_SESSION_TIMEOUT:
+ return false;
+ default:
+ break;
+ }
+
+ return true;
+}
+
+/*
+ * Recover from a possibly aborted transaction state and disable the current
+ * subscription.
+ */
+static ErrorData *
+DisableSubscriptionOnError(MemoryContext mcxt)
+{
+ Relation rel;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Oid subid;
+ Form_pg_subscription subform;
+ ErrorData *edata;
+
+ /*
+ * Clean up from the error and get a fresh transaction in which to
+ * disable the subscription.
+ */
+ MemoryContextSwitchTo(mcxt);
+ edata = CopyErrorData();
+
+ ereport(LOG,
+ (errmsg("logical replication subscription \"%s\" will be disabled due to error: %s",
+ MySubscription->name, edata->message)));
+
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+
+ StartTransactionCommand();
+
+ /* Look up our subscription in the catalogs */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy2(SUBSCRIPTIONNAME, MyDatabaseId,
+ CStringGetDatum(MySubscription->name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("subscription \"%s\" does not exist",
+ MySubscription->name)));
+
+ subform = (Form_pg_subscription) GETSTRUCT(tup);
+ subid = subform->oid;
+ LockSharedObject(SubscriptionRelationId, subid, 0, AccessExclusiveLock);
+
+ /*
+ * We would not be here unless this subscription's disableonerr
+ * field was true when our worker began applying changes, but check
+ * whether that field has changed in the interim.
+ */
+ if (!subform->subdisableonerr)
+ ReThrowError(edata);
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Set the subscription to disabled, and note the reason. */
+ values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(false);
+ replaces[Anum_pg_subscription_subenabled - 1] = true;
+
+ /* Update the catalog */
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+ heap_freetuple(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ CommitTransactionCommand();
+
+ return edata;
+}
+
+/*
* Send a Standby Status Update message to server.
*
* 'recvpos' is the latest LSN we've received data to, force is set if we need
@@ -3336,6 +3455,9 @@ ApplyWorkerMain(Datum main_arg)
char *myslotname;
WalRcvStreamOptions options;
int server_version;
+ bool disable_subscription = false;
+ MemoryContext ecxt;
+ ErrorData *errdata;
/* Attach to slot */
logicalrep_worker_attach(worker_slot);
@@ -3437,8 +3559,8 @@ ApplyWorkerMain(Datum main_arg)
}
PG_CATCH();
{
- MemoryContext ecxt = MemoryContextSwitchTo(cctx);
- ErrorData *errdata = CopyErrorData();
+ ecxt = MemoryContextSwitchTo(cctx);
+ errdata = CopyErrorData();
/*
* Report the table sync error. There is no corresponding message
@@ -3450,11 +3572,26 @@ ApplyWorkerMain(Datum main_arg)
0, /* message type */
InvalidTransactionId,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- PG_RE_THROW();
+
+ /* Decide whether or not we disable this subscription */
+ if (MySubscription->disableonerr &&
+ IsSubscriptionDisablingError())
+ disable_subscription = true;
+ else
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
+ /* If we caught an error above, disable the subscription */
+ if (disable_subscription)
+ {
+ ReThrowError(DisableSubscriptionOnError(cctx));
+ MemoryContextSwitchTo(ecxt);
+ }
+
/* allocate slot name in long-lived context */
myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
@@ -3580,8 +3717,8 @@ ApplyWorkerMain(Datum main_arg)
/* report the apply error */
if (apply_error_callback_arg.command != 0)
{
- MemoryContext ecxt = MemoryContextSwitchTo(cctx);
- ErrorData *errdata = CopyErrorData();
+ ecxt = MemoryContextSwitchTo(cctx);
+ errdata = CopyErrorData();
pgstat_report_subworker_error(MyLogicalRepWorker->subid,
MyLogicalRepWorker->relid,
@@ -3591,13 +3728,25 @@ ApplyWorkerMain(Datum main_arg)
apply_error_callback_arg.command,
apply_error_callback_arg.remote_xid,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- }
- PG_RE_THROW();
+ if (MySubscription->disableonerr &&
+ IsSubscriptionDisablingError())
+ disable_subscription = true;
+ else
+ {
+ PG_RE_THROW();
+ MemoryContextSwitchTo(ecxt);
+ }
+ }
}
PG_END_TRY();
+ if (disable_subscription)
+ {
+ ReThrowError(DisableSubscriptionOnError(cctx));
+ MemoryContextSwitchTo(ecxt);
+ }
+
proc_exit(0);
}
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 2106149..ff50caa 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -60,6 +60,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subdisableonerr; /* True if apply errors should
+ * disable the subscription upon error */
+
bool subbinary; /* True if the subscription wants the
* publisher to send data in binary */
@@ -99,6 +102,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool disableonerr; /* Whether errors automatically disable */
bool binary; /* Indicates if the subscription wants data in
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 9467a19..b830be4 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -2586,6 +2586,46 @@ sub wait_for_slot_catchup
return;
}
+=pot
+
+=item $node->wait_for_subscriptions($dbname, @subcriptions)
+
+Wait for the named subscriptions to catch up or to be disabled.
+
+=cut
+
+sub wait_for_subscriptions
+{
+ my ($self, $dbname, @subscriptions) = @_;
+
+ # Unique-ify the subscriptions passed by the caller
+ my %unique = map { $_ => 1 } @subscriptions;
+ my @unique = sort keys %unique;
+ my $unique_count = scalar(@unique);
+
+ # Construct a SQL list from the unique subscription names
+ my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
+ my $sublist = join(', ', map { "'$_'" } @escaped);
+
+ my $polling_sql = qq(
+ SELECT COUNT(1) = $unique_count FROM
+ (SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ LEFT JOIN pg_catalog.pg_subscription_rel sr
+ ON sr.srsubid = s.oid
+ WHERE (sr IS NULL OR sr.srsubstate IN ('s', 'r'))
+ AND s.subname IN ($sublist)
+ AND s.subenabled IS TRUE
+ UNION
+ SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ WHERE s.subname IN ($sublist)
+ AND s.subenabled IS FALSE
+ ) AS synced_or_disabled
+ );
+ return $self->poll_query_until($dbname, $polling_sql);
+}
+
=pod
=item $node->query_hash($dbname, $query, @columns)
diff --git a/src/test/subscription/t/027_disable_on_error.pl b/src/test/subscription/t/027_disable_on_error.pl
new file mode 100644
index 0000000..e18ce0d
--- /dev/null
+++ b/src/test/subscription/t/027_disable_on_error.pl
@@ -0,0 +1,192 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+# use PostgresNode;
+# use TestLib;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+
+my @schemas = qw(s1 s2);
+my ($schema, $cmd);
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+# Create identical schema, table and index on both the publisher and
+# subscriber
+#
+for $schema (@schemas)
+{
+ $cmd = qq(
+CREATE SCHEMA $schema;
+CREATE TABLE $schema.tbl (i INT);
+ALTER TABLE $schema.tbl REPLICA IDENTITY FULL;
+CREATE INDEX ${schema}_tbl_idx ON $schema.tbl(i));
+ $node_publisher->safe_psql('postgres', $cmd);
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Create non-unique data in both schemas on the publisher.
+#
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (1), (1), (1));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Create an additional unique index in schema s1 on the subscriber only. When
+# we create subscriptions, below, this should cause subscription "s1" on the
+# subscriber to fail during initial synchronization and to get automatically
+# disabled.
+#
+$cmd = qq(CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Create publications and subscriptions linking the schemas on
+# the publisher with those on the subscriber. This tests that the
+# uniqueness violations cause subscription "s1" to fail during
+# initial synchronization.
+#
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+for $schema (@schemas)
+{
+ # Create the publication for this table
+ $cmd = qq(
+CREATE PUBLICATION $schema FOR TABLE $schema.tbl);
+ $node_publisher->safe_psql('postgres', $cmd);
+
+ # Create the subscription for this table
+ $cmd = qq(
+CREATE SUBSCRIPTION $schema
+ CONNECTION '$publisher_connstr'
+ PUBLICATION $schema
+ WITH (disable_on_error = true));
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Wait for the initial subscription synchronizations to finish or fail.
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT i, COUNT(*) FROM s2.tbl GROUP BY i);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "1|3",
+ "subscription s2 replicated initial data");
+
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+#
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (2));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that the
+# problems encountered by subscription "s1" do not cause subscription "s2" to
+# get stuck.
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should still be disabled and have replicated no data
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 still disabled");
+
+# Subscription "s2" should still be enabled and have replicated all changes
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s2 replicated data");
+
+# Drop the unique index on "s1" which caused the subscription to be disabled
+#
+$cmd = qq(DROP INDEX s1.s1_tbl_unique);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Re-enable the subscription "s1"
+#
+$cmd = q(ALTER SUBSCRIPTION s1 ENABLE);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Wait for the data to replicate
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check that we have the new data in s1.tbl
+#
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s1.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s1 replicated data");
+
+# Delete the data from the subscriber only, and recreate the unique index
+#
+$cmd = q(
+DELETE FROM s1.tbl;
+CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Add more non-unique data to the publisher
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (3), (3), (3));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that
+# uniqueness violations encountered during replication cause s1 to be disabled.
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "3|7",
+ "subscription s2 replicated additional data");
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
2.2.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-08 13:14 vignesh C <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: vignesh C @ 2021-11-08 13:14 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Nov 2, 2021 at 4:12 PM [email protected]
<[email protected]> wrote:
>
> On Monday, June 28, 2021 1:47 PM Masahiko Sawada <[email protected]> wrote:
> > On Mon, Jun 21, 2021 at 11:26 AM Mark Dilger
> > <[email protected]> wrote:
> > > > On Jun 20, 2021, at 7:17 PM, Masahiko Sawada
> > <[email protected]> wrote:
> > > >
> > > > I will submit the patch.
> > >
> > > Great, thanks!
> >
> > I've submitted the patches on that thread[1]. There are three patches:
> > skipping the transaction on the subscriber side, reporting error details in the
> > errcontext, and reporting the error details to the stats collector. Feedback is
> > very welcome.
> >
> > [1]
> > https://www.postgresql.org/message-id/CAD21AoBU4jGEO6AXcykQ9y7tat0R
> > rB5--8ZoJgfcj%2BLPs7nFZQ%40mail.gmail.com
> Hi, thanks Sawada-san for keep updating the skip xid patch in the thread.
>
> This thread has stopped since the patch submission.
> I've rebased the 'disable_on_error' option
> so that it can be applied on top of skip xid shared in [1].
> I've written Mark Dilger as the original author in the commit message.
>
> This patch is simply rebased to reactive this thread.
> So there are still pending item to discuss for example,
> how we should deal with multiple errors of several table sync workers.
>
> I extracted only 'disable_on_error' option
> because the skip xid and the latest error message fulfill the motivation
> to make it easy to write TAP tests already I felt.
>
Thanks for the updated patch. Please create a Commitfest entry for
this. It will help to have a look at CFBot results for the patch, also
if required rebase and post a patch on top of Head.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-10 01:26 [email protected] <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: [email protected] @ 2021-11-10 01:26 UTC (permalink / raw)
To: 'vignesh C' <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Monday, November 8, 2021 10:15 PM vignesh C <[email protected]> wrote:
> Thanks for the updated patch. Please create a Commitfest entry for this. It will
> help to have a look at CFBot results for the patch, also if required rebase and
> post a patch on top of Head.
As requested, created a new entry for this - [1]
FYI: the skip xid patch has been updated to v20 in [2]
but the v3 for disable_on_error is not affected by this update
and still applicable with no regression.
[1] - https://commitfest.postgresql.org/36/3407/
[2] - https://www.postgresql.org/message-id/CAD21AoAT42mhcqeB1jPfRL1%2BEUHbZk8MMY_fBgsyZvJeKNpG%2Bw%40mail...
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-10 04:22 Greg Nancarrow <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: Greg Nancarrow @ 2021-11-10 04:22 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Nov 10, 2021 at 12:26 PM [email protected]
<[email protected]> wrote:
>
> On Monday, November 8, 2021 10:15 PM vignesh C <[email protected]> wrote:
> > Thanks for the updated patch. Please create a Commitfest entry for this. It will
> > help to have a look at CFBot results for the patch, also if required rebase and
> > post a patch on top of Head.
> As requested, created a new entry for this - [1]
>
> FYI: the skip xid patch has been updated to v20 in [2]
> but the v3 for disable_on_error is not affected by this update
> and still applicable with no regression.
>
> [1] - https://commitfest.postgresql.org/36/3407/
> [2] - https://www.postgresql.org/message-id/CAD21AoAT42mhcqeB1jPfRL1%2BEUHbZk8MMY_fBgsyZvJeKNpG%2Bw%40mail...
>
I had a look at this patch and have a couple of initial review
comments for some issues I spotted:
src/backend/commands/subscriptioncmds.c
(1) bad array entry assignment
The following code block added by the patch assigns
"values[Anum_pg_subscription_subdisableonerr - 1]" twice, resulting in
it being always set to true, rather than the specified option value:
+ if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR))
+ {
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = BoolGetDatum(opts.disableonerr);
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = true;
+ }
The 2nd line is meant to instead be
"replaces[Anum_pg_subscription_subdisableonerr - 1] = true".
(compare to handling for other similar options)
src/backend/replication/logical/worker.c
(2) unreachable code?
In the patch code there seems to be some instances of unreachable code
after re-throwing errors:
e.g.
+ /* If we caught an error above, disable the subscription */
+ if (disable_subscription)
+ {
+ ReThrowError(DisableSubscriptionOnError(cctx));
+ MemoryContextSwitchTo(ecxt);
+ }
+ else
+ {
+ PG_RE_THROW();
+ MemoryContextSwitchTo(ecxt);
+ }
+ if (disable_subscription)
+ {
+ ReThrowError(DisableSubscriptionOnError(cctx));
+ MemoryContextSwitchTo(ecxt);
+ }
I'm guessing it was intended to do the "MemoryContextSwitch(ecxt);"
before re-throwing (?), but it's not really clear, as in the 1st and
3rd cases, the DisableSubscriptionOnError() calls anyway immediately
switch the memory context to cctx.
Regards,
Greg Nancarrow
Fujitsu Australia
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-10 04:31 Greg Nancarrow <[email protected]>
parent: Greg Nancarrow <[email protected]>
1 sibling, 0 replies; 74+ messages in thread
From: Greg Nancarrow @ 2021-11-10 04:31 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Nov 10, 2021 at 3:22 PM Greg Nancarrow <[email protected]> wrote:
>
> I had a look at this patch and have a couple of initial review
> comments for some issues I spotted:
>
Incidentally, I found that the v3 patch only applies after the skip xid v20
patch [1] has been applied.
[2] -
https://www.postgresql.org/message-id/CAD21AoAT42mhcqeB1jPfRL1%2BEUHbZk8MMY_fBgsyZvJeKNpG%2Bw%40mail...
Regards,
Greg Nancarrow
Fujitsu Australia
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-11 09:20 [email protected] <[email protected]>
parent: Greg Nancarrow <[email protected]>
1 sibling, 2 replies; 74+ messages in thread
From: [email protected] @ 2021-11-11 09:20 UTC (permalink / raw)
To: 'Greg Nancarrow' <[email protected]>; +Cc: vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wednesday, November 10, 2021 1:23 PM Greg Nancarrow <[email protected]> wrote:
> On Wed, Nov 10, 2021 at 12:26 PM [email protected]
> <[email protected]> wrote:
> >
> > On Monday, November 8, 2021 10:15 PM vignesh C <[email protected]>
> wrote:
> > > Thanks for the updated patch. Please create a Commitfest entry for
> > > this. It will help to have a look at CFBot results for the patch,
> > > also if required rebase and post a patch on top of Head.
> > As requested, created a new entry for this - [1]
> >
> > FYI: the skip xid patch has been updated to v20 in [2] but the v3 for
> > disable_on_error is not affected by this update and still applicable
> > with no regression.
> >
> > [1] - https://commitfest.postgresql.org/36/3407/
> > [2] -
> >
> https://www.postgresql.org/message-id/CAD21AoAT42mhcqeB1jPfRL1%2B
> EUHbZ
> > k8MMY_fBgsyZvJeKNpG%2Bw%40mail.gmail.com
>
> I had a look at this patch and have a couple of initial review comments for some
> issues I spotted:
Thank you for checking it.
> src/backend/commands/subscriptioncmds.c
> (1) bad array entry assignment
> The following code block added by the patch assigns
> "values[Anum_pg_subscription_subdisableonerr - 1]" twice, resulting in it
> being always set to true, rather than the specified option value:
>
> + if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR)) {
> + values[Anum_pg_subscription_subdisableonerr - 1]
> + = BoolGetDatum(opts.disableonerr);
> + values[Anum_pg_subscription_subdisableonerr - 1]
> + = true;
> + }
>
> The 2nd line is meant to instead be
> "replaces[Anum_pg_subscription_subdisableonerr - 1] = true".
> (compare to handling for other similar options)
Oops, fixed.
> src/backend/replication/logical/worker.c
> (2) unreachable code?
> In the patch code there seems to be some instances of unreachable code after
> re-throwing errors:
>
> e.g.
>
> + /* If we caught an error above, disable the subscription */ if
> + (disable_subscription) {
> + ReThrowError(DisableSubscriptionOnError(cctx));
> + MemoryContextSwitchTo(ecxt);
> + }
>
> + else
> + {
> + PG_RE_THROW();
> + MemoryContextSwitchTo(ecxt);
> + }
>
>
> + if (disable_subscription)
> + {
> + ReThrowError(DisableSubscriptionOnError(cctx));
> + MemoryContextSwitchTo(ecxt);
> + }
>
> I'm guessing it was intended to do the "MemoryContextSwitch(ecxt);"
> before re-throwing (?), but it's not really clear, as in the 1st and 3rd cases, the
> DisableSubscriptionOnError() calls anyway immediately switch the memory
> context to cctx.
You are right I think.
Fixed based on an idea below.
After an error happens, for some additional work
(e.g. to report the stats of table sync/apply worker
by pgstat_report_subworker_error() or
to update the catalog by DisableSubscriptionOnError())
restore the memory context that is used before the error (cctx)
and save the old memory context of error (ecxt). Then,
do the additional work and switch the memory context to the ecxt
just before the rethrow. As you described,
in contrast to PG_RE_THROW, DisableSubscriptionOnError() changes
the memory context immediatedly at the top of it,
so for this case, I don't call the MemoryContextSwitchTo().
Another important thing as my modification
is a case when LogicalRepApplyLoop failed and
apply_error_callback_arg.command == 0. In the original
patch of skip xid, it just calls PG_RE_THROW()
but my previous v3 codes missed this macro in this case.
Therefore, I've fixed this part as well.
C codes are checked by pgindent.
Note that this depends on the v20 skip xide patch in [1]
[1] - https://www.postgresql.org/message-id/CAD21AoAT42mhcqeB1jPfRL1%2BEUHbZk8MMY_fBgsyZvJeKNpG%2Bw%40mail...
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v4-0001-Optionally-disabling-subscriptions-on-error.patch (22.9K, ../../TYCPR01MB8373D8A622ED51CDEF3DAF5DED949@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v4-0001-Optionally-disabling-subscriptions-on-error.patch)
download | inline diff:
From 3276816ab453883d0fcda87c84eed5c07a6cf136 Mon Sep 17 00:00:00 2001
From: Osumi Takamichi <[email protected]>
Date: Thu, 11 Nov 2021 08:54:31 +0000
Subject: [PATCH v4] Optionally disabling subscriptions on error
Logical replication apply workers for a subscription can easily get
stuck in an infinite loop of attempting to apply a change,
triggering an error (such as a constraint violation), exiting with
an error written to the subscription worker log, and restarting.
To partially remedy the situation, adding a new
subscription_parameter named 'disable_on_error'. To be consistent
with old behavior, the parameter defaults to false. When true, the
apply worker catches errors thrown, and for errors that are deemed
not to be transient, disables the subscription in order to break the
loop. The error is still also written to the logs.
Proposed and written originally by Mark Dilger.
---
doc/src/sgml/ref/create_subscription.sgml | 12 ++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 27 +++-
src/backend/replication/logical/launcher.c | 1 +
src/backend/replication/logical/worker.c | 157 ++++++++++++++++++-
src/include/catalog/pg_subscription.h | 4 +
src/test/perl/PostgreSQL/Test/Cluster.pm | 40 +++++
src/test/subscription/t/027_disable_on_error.pl | 192 ++++++++++++++++++++++++
9 files changed, 428 insertions(+), 8 deletions(-)
create mode 100644 src/test/subscription/t/027_disable_on_error.pl
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 990a41f..ef12277 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -142,6 +142,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</varlistentry>
<varlistentry>
+ <term><literal>disable_on_error</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription should be automatically disabled
+ if replicating data from the publisher triggers non-transicent errors
+ such as referential integrity or permissions errors. The default is
+ <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>enabled</literal> (<type>boolean</type>)</term>
<listitem>
<para>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 25021e2..fcace32 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->binary = subform->subbinary;
sub->stream = subform->substream;
sub->twophasestate = subform->subtwophasestate;
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index fece48a..ac7dd0a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1258,7 +1258,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
+GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subdisableonerr, subbinary,
substream, subtwophasestate, subslotname, subsynccommit, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 18962b9..53a65ad 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -61,6 +61,7 @@
#define SUBOPT_BINARY 0x00000080
#define SUBOPT_STREAMING 0x00000100
#define SUBOPT_TWOPHASE_COMMIT 0x00000200
+#define SUBOPT_DISABLE_ON_ERR 0x00000400
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -82,6 +83,7 @@ typedef struct SubOpts
bool binary;
bool streaming;
bool twophase;
+ bool disableonerr;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -129,6 +131,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->streaming = false;
if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT))
opts->twophase = false;
+ if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
+ opts->disableonerr = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -248,6 +252,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_TWOPHASE_COMMIT;
opts->twophase = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR) &&
+ strcmp(defel->defname, "disable_on_error") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DISABLE_ON_ERR))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DISABLE_ON_ERR;
+ opts->disableonerr = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -397,7 +410,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
supported_opts = (SUBOPT_CONNECT | SUBOPT_ENABLED | SUBOPT_CREATE_SLOT |
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT);
+ SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
+ SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -465,6 +479,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled);
+ values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary);
values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming);
values[Anum_pg_subscription_subtwophasestate - 1] =
@@ -871,11 +886,19 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
{
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING);
+ SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
+ if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR))
+ {
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = BoolGetDatum(opts.disableonerr);
+ replaces[Anum_pg_subscription_subdisableonerr - 1]
+ = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_SLOT_NAME))
{
/*
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 3fb4caa..febfc4d 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -132,6 +132,7 @@ get_subscription_list(void)
sub->dbid = subform->subdbid;
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->name = pstrdup(NameStr(subform->subname));
/* We don't fill fields we are not interested in. */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index e2a929b..5010b5e 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -136,6 +136,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/partition.h"
#include "catalog/pg_inherits.h"
@@ -334,6 +335,7 @@ static void apply_spooled_messages(TransactionId xid, XLogRecPtr lsn);
static void apply_error_callback(void *arg);
static inline void set_apply_error_context_xact(TransactionId xid, TimestampTz ts);
static inline void reset_apply_error_context_info(void);
+static bool IsSubscriptionDisablingError(void);
/*
* Should this worker apply changes for given relation.
@@ -2752,6 +2754,123 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/*
+ * Errors which are transient, network protocol related, or resource exhaustion
+ * related, should not disable a subscription. These may clear up without user
+ * intervention in the subscription, schema, or data being replicated.
+ */
+static bool
+IsSubscriptionDisablingError(void)
+{
+ switch (geterrcode())
+ {
+ case ERRCODE_CONNECTION_EXCEPTION:
+ case ERRCODE_CONNECTION_DOES_NOT_EXIST:
+ case ERRCODE_CONNECTION_FAILURE:
+ case ERRCODE_SQLCLIENT_UNABLE_TO_ESTABLISH_SQLCONNECTION:
+ case ERRCODE_SQLSERVER_REJECTED_ESTABLISHMENT_OF_SQLCONNECTION:
+ case ERRCODE_TRANSACTION_RESOLUTION_UNKNOWN:
+ case ERRCODE_PROTOCOL_VIOLATION:
+ case ERRCODE_INSUFFICIENT_RESOURCES:
+ case ERRCODE_DISK_FULL:
+ case ERRCODE_OUT_OF_MEMORY:
+ case ERRCODE_TOO_MANY_CONNECTIONS:
+ case ERRCODE_CONFIGURATION_LIMIT_EXCEEDED:
+ case ERRCODE_PROGRAM_LIMIT_EXCEEDED:
+ case ERRCODE_STATEMENT_TOO_COMPLEX:
+ case ERRCODE_TOO_MANY_COLUMNS:
+ case ERRCODE_TOO_MANY_ARGUMENTS:
+ case ERRCODE_OPERATOR_INTERVENTION:
+ case ERRCODE_QUERY_CANCELED:
+ case ERRCODE_ADMIN_SHUTDOWN:
+ case ERRCODE_CRASH_SHUTDOWN:
+ case ERRCODE_CANNOT_CONNECT_NOW:
+ case ERRCODE_DATABASE_DROPPED:
+ case ERRCODE_IDLE_SESSION_TIMEOUT:
+ return false;
+ default:
+ break;
+ }
+
+ return true;
+}
+
+/*
+ * Recover from a possibly aborted transaction state and disable the current
+ * subscription.
+ */
+static ErrorData *
+DisableSubscriptionOnError(MemoryContext mcxt)
+{
+ Relation rel;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Oid subid;
+ Form_pg_subscription subform;
+ ErrorData *edata;
+
+ /*
+ * Clean up from the error and get a fresh transaction in which to
+ * disable the subscription.
+ */
+ MemoryContextSwitchTo(mcxt);
+ edata = CopyErrorData();
+
+ ereport(LOG,
+ (errmsg("logical replication subscription \"%s\" will be disabled due to error: %s",
+ MySubscription->name, edata->message)));
+
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+
+ StartTransactionCommand();
+
+ /* Look up our subscription in the catalogs */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy2(SUBSCRIPTIONNAME, MyDatabaseId,
+ CStringGetDatum(MySubscription->name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("subscription \"%s\" does not exist",
+ MySubscription->name)));
+
+ subform = (Form_pg_subscription) GETSTRUCT(tup);
+ subid = subform->oid;
+ LockSharedObject(SubscriptionRelationId, subid, 0, AccessExclusiveLock);
+
+ /*
+ * We would not be here unless this subscription's disableonerr
+ * field was true when our worker began applying changes, but check
+ * whether that field has changed in the interim.
+ */
+ if (!subform->subdisableonerr)
+ ReThrowError(edata);
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Set the subscription to disabled, and note the reason. */
+ values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(false);
+ replaces[Anum_pg_subscription_subenabled - 1] = true;
+
+ /* Update the catalog */
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+ heap_freetuple(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ CommitTransactionCommand();
+
+ return edata;
+}
+
+/*
* Send a Standby Status Update message to server.
*
* 'recvpos' is the latest LSN we've received data to, force is set if we need
@@ -3336,6 +3455,7 @@ ApplyWorkerMain(Datum main_arg)
char *myslotname;
WalRcvStreamOptions options;
int server_version;
+ bool disable_subscription = false;
/* Attach to slot */
logicalrep_worker_attach(worker_slot);
@@ -3450,11 +3570,23 @@ ApplyWorkerMain(Datum main_arg)
0, /* message type */
InvalidTransactionId,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- PG_RE_THROW();
+
+ /* Decide whether or not we disable this subscription */
+ if (MySubscription->disableonerr &&
+ IsSubscriptionDisablingError())
+ disable_subscription = true;
+ else
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
+ /* If we caught an error above, disable the subscription */
+ if (disable_subscription)
+ ReThrowError(DisableSubscriptionOnError(cctx));
+
/* allocate slot name in long-lived context */
myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
@@ -3591,13 +3723,28 @@ ApplyWorkerMain(Datum main_arg)
apply_error_callback_arg.command,
apply_error_callback_arg.remote_xid,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- }
- PG_RE_THROW();
+ if (MySubscription->disableonerr &&
+ IsSubscriptionDisablingError())
+ disable_subscription = true;
+ else
+ {
+ /*
+ * Some work in error recovery work is done.
+ * Switch to the old memory context.
+ */
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ }
+ else
+ PG_RE_THROW(); /* No need to change the memory context */
}
PG_END_TRY();
+ if (disable_subscription)
+ ReThrowError(DisableSubscriptionOnError(cctx));
+
proc_exit(0);
}
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 2106149..9912b90 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -60,6 +60,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subdisableonerr; /* True if apply errors should disable the
+ * subscription upon error */
+
bool subbinary; /* True if the subscription wants the
* publisher to send data in binary */
@@ -99,6 +102,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool disableonerr; /* Whether errors automatically disable */
bool binary; /* Indicates if the subscription wants data in
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 9467a19..b830be4 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -2586,6 +2586,46 @@ sub wait_for_slot_catchup
return;
}
+=pot
+
+=item $node->wait_for_subscriptions($dbname, @subcriptions)
+
+Wait for the named subscriptions to catch up or to be disabled.
+
+=cut
+
+sub wait_for_subscriptions
+{
+ my ($self, $dbname, @subscriptions) = @_;
+
+ # Unique-ify the subscriptions passed by the caller
+ my %unique = map { $_ => 1 } @subscriptions;
+ my @unique = sort keys %unique;
+ my $unique_count = scalar(@unique);
+
+ # Construct a SQL list from the unique subscription names
+ my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
+ my $sublist = join(', ', map { "'$_'" } @escaped);
+
+ my $polling_sql = qq(
+ SELECT COUNT(1) = $unique_count FROM
+ (SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ LEFT JOIN pg_catalog.pg_subscription_rel sr
+ ON sr.srsubid = s.oid
+ WHERE (sr IS NULL OR sr.srsubstate IN ('s', 'r'))
+ AND s.subname IN ($sublist)
+ AND s.subenabled IS TRUE
+ UNION
+ SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ WHERE s.subname IN ($sublist)
+ AND s.subenabled IS FALSE
+ ) AS synced_or_disabled
+ );
+ return $self->poll_query_until($dbname, $polling_sql);
+}
+
=pod
=item $node->query_hash($dbname, $query, @columns)
diff --git a/src/test/subscription/t/027_disable_on_error.pl b/src/test/subscription/t/027_disable_on_error.pl
new file mode 100644
index 0000000..e18ce0d
--- /dev/null
+++ b/src/test/subscription/t/027_disable_on_error.pl
@@ -0,0 +1,192 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+# use PostgresNode;
+# use TestLib;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+
+my @schemas = qw(s1 s2);
+my ($schema, $cmd);
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+# Create identical schema, table and index on both the publisher and
+# subscriber
+#
+for $schema (@schemas)
+{
+ $cmd = qq(
+CREATE SCHEMA $schema;
+CREATE TABLE $schema.tbl (i INT);
+ALTER TABLE $schema.tbl REPLICA IDENTITY FULL;
+CREATE INDEX ${schema}_tbl_idx ON $schema.tbl(i));
+ $node_publisher->safe_psql('postgres', $cmd);
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Create non-unique data in both schemas on the publisher.
+#
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (1), (1), (1));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Create an additional unique index in schema s1 on the subscriber only. When
+# we create subscriptions, below, this should cause subscription "s1" on the
+# subscriber to fail during initial synchronization and to get automatically
+# disabled.
+#
+$cmd = qq(CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Create publications and subscriptions linking the schemas on
+# the publisher with those on the subscriber. This tests that the
+# uniqueness violations cause subscription "s1" to fail during
+# initial synchronization.
+#
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+for $schema (@schemas)
+{
+ # Create the publication for this table
+ $cmd = qq(
+CREATE PUBLICATION $schema FOR TABLE $schema.tbl);
+ $node_publisher->safe_psql('postgres', $cmd);
+
+ # Create the subscription for this table
+ $cmd = qq(
+CREATE SUBSCRIPTION $schema
+ CONNECTION '$publisher_connstr'
+ PUBLICATION $schema
+ WITH (disable_on_error = true));
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Wait for the initial subscription synchronizations to finish or fail.
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT i, COUNT(*) FROM s2.tbl GROUP BY i);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "1|3",
+ "subscription s2 replicated initial data");
+
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+#
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (2));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that the
+# problems encountered by subscription "s1" do not cause subscription "s2" to
+# get stuck.
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should still be disabled and have replicated no data
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 still disabled");
+
+# Subscription "s2" should still be enabled and have replicated all changes
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s2 replicated data");
+
+# Drop the unique index on "s1" which caused the subscription to be disabled
+#
+$cmd = qq(DROP INDEX s1.s1_tbl_unique);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Re-enable the subscription "s1"
+#
+$cmd = q(ALTER SUBSCRIPTION s1 ENABLE);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Wait for the data to replicate
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check that we have the new data in s1.tbl
+#
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s1.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s1 replicated data");
+
+# Delete the data from the subscriber only, and recreate the unique index
+#
+$cmd = q(
+DELETE FROM s1.tbl;
+CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Add more non-unique data to the publisher
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (3), (3), (3));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that
+# uniqueness violations encountered during replication cause s1 to be disabled.
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "3|7",
+ "subscription s2 replicated additional data");
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
2.2.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-12 04:08 vignesh C <[email protected]>
parent: [email protected] <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: vignesh C @ 2021-11-12 04:08 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Nov 11, 2021 at 2:50 PM [email protected]
<[email protected]> wrote:
>
> On Wednesday, November 10, 2021 1:23 PM Greg Nancarrow <[email protected]> wrote:
> > On Wed, Nov 10, 2021 at 12:26 PM [email protected]
> > <[email protected]> wrote:
> > >
> > > On Monday, November 8, 2021 10:15 PM vignesh C <[email protected]>
> > wrote:
> > > > Thanks for the updated patch. Please create a Commitfest entry for
> > > > this. It will help to have a look at CFBot results for the patch,
> > > > also if required rebase and post a patch on top of Head.
> > > As requested, created a new entry for this - [1]
> > >
> > > FYI: the skip xid patch has been updated to v20 in [2] but the v3 for
> > > disable_on_error is not affected by this update and still applicable
> > > with no regression.
> > >
> > > [1] - https://commitfest.postgresql.org/36/3407/
> > > [2] -
> > >
> > https://www.postgresql.org/message-id/CAD21AoAT42mhcqeB1jPfRL1%2B
> > EUHbZ
> > > k8MMY_fBgsyZvJeKNpG%2Bw%40mail.gmail.com
> >
> > I had a look at this patch and have a couple of initial review comments for some
> > issues I spotted:
> Thank you for checking it.
>
>
> > src/backend/commands/subscriptioncmds.c
> > (1) bad array entry assignment
> > The following code block added by the patch assigns
> > "values[Anum_pg_subscription_subdisableonerr - 1]" twice, resulting in it
> > being always set to true, rather than the specified option value:
> >
> > + if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR)) {
> > + values[Anum_pg_subscription_subdisableonerr - 1]
> > + = BoolGetDatum(opts.disableonerr);
> > + values[Anum_pg_subscription_subdisableonerr - 1]
> > + = true;
> > + }
> >
> > The 2nd line is meant to instead be
> > "replaces[Anum_pg_subscription_subdisableonerr - 1] = true".
> > (compare to handling for other similar options)
> Oops, fixed.
>
> > src/backend/replication/logical/worker.c
> > (2) unreachable code?
> > In the patch code there seems to be some instances of unreachable code after
> > re-throwing errors:
> >
> > e.g.
> >
> > + /* If we caught an error above, disable the subscription */ if
> > + (disable_subscription) {
> > + ReThrowError(DisableSubscriptionOnError(cctx));
> > + MemoryContextSwitchTo(ecxt);
> > + }
> >
> > + else
> > + {
> > + PG_RE_THROW();
> > + MemoryContextSwitchTo(ecxt);
> > + }
> >
> >
> > + if (disable_subscription)
> > + {
> > + ReThrowError(DisableSubscriptionOnError(cctx));
> > + MemoryContextSwitchTo(ecxt);
> > + }
> >
> > I'm guessing it was intended to do the "MemoryContextSwitch(ecxt);"
> > before re-throwing (?), but it's not really clear, as in the 1st and 3rd cases, the
> > DisableSubscriptionOnError() calls anyway immediately switch the memory
> > context to cctx.
> You are right I think.
> Fixed based on an idea below.
>
> After an error happens, for some additional work
> (e.g. to report the stats of table sync/apply worker
> by pgstat_report_subworker_error() or
> to update the catalog by DisableSubscriptionOnError())
> restore the memory context that is used before the error (cctx)
> and save the old memory context of error (ecxt). Then,
> do the additional work and switch the memory context to the ecxt
> just before the rethrow. As you described,
> in contrast to PG_RE_THROW, DisableSubscriptionOnError() changes
> the memory context immediatedly at the top of it,
> so for this case, I don't call the MemoryContextSwitchTo().
>
> Another important thing as my modification
> is a case when LogicalRepApplyLoop failed and
> apply_error_callback_arg.command == 0. In the original
> patch of skip xid, it just calls PG_RE_THROW()
> but my previous v3 codes missed this macro in this case.
> Therefore, I've fixed this part as well.
>
> C codes are checked by pgindent.
>
> Note that this depends on the v20 skip xide patch in [1]
>
Thanks for the updated patch, Few comments:
1) tab completion should be added for disable_on_error:
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
"enabled", "slot_name", "streaming",
"synchronous_commit", "two_phase");
2) disable_on_error is supported by alter subscription, the same
should be documented:
@ -871,11 +886,19 @@ AlterSubscription(ParseState *pstate,
AlterSubscriptionStmt *stmt,
{
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
-
SUBOPT_STREAMING);
+
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate,
stmt->options,
supported_opts, &opts);
+ if (IsSet(opts.specified_opts,
SUBOPT_DISABLE_ON_ERR))
+ {
+
values[Anum_pg_subscription_subdisableonerr - 1]
+ =
BoolGetDatum(opts.disableonerr);
+
replaces[Anum_pg_subscription_subdisableonerr - 1]
+ = true;
+ }
+
3) Describe subscriptions (dRs+) should include displaying of disableonerr:
\dRs+ sub1
List of subscriptions
Name | Owner | Enabled | Publication | Binary | Streaming | Two
phase commit | Synchronous commit | Conninfo
------+---------+---------+-------------+--------+-----------+------------------+--------------------+---------------------------
sub1 | vignesh | t | {pub1} | f | f | d
| off | dbname=postgres port=5432
(1 row)
4) I felt transicent should be transient, might be a typo:
+ Specifies whether the subscription should be automatically disabled
+ if replicating data from the publisher triggers non-transicent errors
+ such as referential integrity or permissions errors. The default is
+ <literal>false</literal>.
5) The commented use PostgresNode and use TestLib can be removed:
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+# use PostgresNode;
+# use TestLib;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
Regards,
Vignesh
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-12 04:48 Greg Nancarrow <[email protected]>
parent: [email protected] <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Greg Nancarrow @ 2021-11-12 04:48 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Nov 11, 2021 at 8:20 PM [email protected]
<[email protected]> wrote:
>
> C codes are checked by pgindent.
>
> Note that this depends on the v20 skip xide patch in [1]
>
Some comments on the v4 patch:
(1) Patch subject
I think the patch subject should say "disable" instead of "disabling":
Optionally disable subscriptions on error
doc/src/sgml/ref/create_subscription.sgml
(2) spelling mistake
+ if replicating data from the publisher triggers non-transicent errors
non-transicent -> non-transient
(I notice Vignesh also pointed this out)
src/backend/replication/logical/worker.c
(3) calling geterrcode()
The new IsSubscriptionDisablingError() function calls geterrcode().
According to the function comment for geterrcode(), it is only
intended for use in error callbacks.
Instead of calling geterrcode(), couldn't the ErrorData from PG_CATCH
block be passed to IsSubscriptionDisablingError() instead (from which
it can get the sqlerrcode)?
(4) DisableSubscriptionOnError
DisableSubscriptionOnError() is again calling MemoryContextSwitch()
and CopyErrorData().
I think the ErrorData from the PG_CATCH block could simply be passed
to DisableSubscriptionOnError() instead of the memory-context, and the
existing MemoryContextSwitch() and CopyErrorData() calls could be
removed from it.
AFAICS, applying (3) and (4) above would make the code a lot cleaner.
Regards,
Greg Nancarrow
Fujitsu Australia
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-16 07:53 [email protected] <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: [email protected] @ 2021-11-16 07:53 UTC (permalink / raw)
To: 'vignesh C' <[email protected]>; +Cc: Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Friday, November 12, 2021 1:09 PM vignesh C <[email protected]> wrote:
> On Thu, Nov 11, 2021 at 2:50 PM [email protected]
> <[email protected]> wrote:
> Thanks for the updated patch, Few comments:
> 1) tab completion should be added for disable_on_error:
> /* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */ else if
> (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
> COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
> "enabled", "slot_name", "streaming",
> "synchronous_commit", "two_phase");
Fixed.
> 2) disable_on_error is supported by alter subscription, the same should be
> documented:
> @ -871,11 +886,19 @@ AlterSubscription(ParseState *pstate,
> AlterSubscriptionStmt *stmt,
> {
> supported_opts = (SUBOPT_SLOT_NAME |
>
> SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
> -
> SUBOPT_STREAMING);
> +
> SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR);
>
> parse_subscription_options(pstate,
> stmt->options,
>
> supported_opts, &opts);
>
> + if (IsSet(opts.specified_opts,
> SUBOPT_DISABLE_ON_ERR))
> + {
> +
> values[Anum_pg_subscription_subdisableonerr - 1]
> + =
> BoolGetDatum(opts.disableonerr);
> +
> replaces[Anum_pg_subscription_subdisableonerr - 1]
> + = true;
> + }
> +
Fixed the documentation. Also, add one test for alter subscription.
> 3) Describe subscriptions (dRs+) should include displaying of disableonerr:
> \dRs+ sub1
> List of subscriptions
> Name | Owner | Enabled | Publication | Binary | Streaming | Two
> phase commit | Synchronous commit | Conninfo
> ------+---------+---------+-------------+--------+-----------+----------
> --------+--------------------+---------------------------
> sub1 | vignesh | t | {pub1} | f | f | d
> | off | dbname=postgres port=5432
> (1 row)
Fixed.
> 4) I felt transicent should be transient, might be a typo:
> + Specifies whether the subscription should be automatically
> disabled
> + if replicating data from the publisher triggers non-transicent errors
> + such as referential integrity or permissions errors. The default is
> + <literal>false</literal>.
Fixed.
> 5) The commented use PostgresNode and use TestLib can be removed:
> +# Test of logical replication subscription self-disabling feature use
> +strict; use warnings; # use PostgresNode; # use TestLib; use
> +PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More
> +tests => 10;
Removed.
Also, my colleague Greg provided an offlist patch to me and
I've incorporated his suggested modifications into this version.
So, I noted his name as a coauthor.
C codes are checked by pgindent again.
This v5 depends on v23 skip xid in [1].
[1] - https://www.postgresql.org/message-id/CAD21AoA5jupM6O%3DpYsyfaxQ1aMX-en8%3DQNgpW6KfXsg7_CS0CQ%40mail...
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v5-0001-Optionally-disable-subscriptions-on-error.patch (48.0K, ../../TYCPR01MB8373771371B31E1E6CC74B0AED999@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v5-0001-Optionally-disable-subscriptions-on-error.patch)
download | inline diff:
From d197478029163c511516b732cb66a474463eb3d7 Mon Sep 17 00:00:00 2001
From: Osumi Takamichi <[email protected]>
Date: Tue, 16 Nov 2021 07:27:02 +0000
Subject: [PATCH v5] Optionally disable subscriptions on error
Logical replication apply workers for a subscription can easily get
stuck in an infinite loop of attempting to apply a change,
triggering an error (such as a constraint violation), exiting with
an error written to the subscription worker log, and restarting.
To partially remedy the situation, adding a new
subscription_parameter named 'disable_on_error'. To be consistent
with old behavior, the parameter defaults to false. When true, the
apply worker catches errors thrown, and for errors that are deemed
not to be transient, disables the subscription in order to break the
loop. The error is still also written to the logs.
Proposed and written originally by Mark Dilger
Taken over by Osumi Takamichi, Greg Nancarrow
Reviewed by Greg Nancarrow, Vignesh C
Discussion : https://www.postgresql.org/message-id/DB35438F-9356-4841-89A0-412709EBD3AB%40enterprisedb.com
---
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 ++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 27 +++-
src/backend/replication/logical/launcher.c | 1 +
src/backend/replication/logical/worker.c | 157 +++++++++++++++++++-
src/bin/psql/describe.c | 10 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 4 +
src/test/perl/PostgreSQL/Test/Cluster.pm | 40 +++++
src/test/regress/expected/subscription.out | 119 +++++++++------
src/test/regress/sql/subscription.sql | 14 ++
src/test/subscription/t/027_disable_on_error.pl | 190 ++++++++++++++++++++++++
14 files changed, 519 insertions(+), 66 deletions(-)
create mode 100644 src/test/subscription/t/027_disable_on_error.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 0b027cc..72b2469 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -201,8 +201,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
information. The parameters that can be altered
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
- <literal>binary</literal>, and
- <literal>streaming</literal>.
+ <literal>binary</literal>,<literal>streaming</literal>
+ <literal>disable_on_err</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 990a41f..0d90d52 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -142,6 +142,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</varlistentry>
<varlistentry>
+ <term><literal>disable_on_error</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription should be automatically disabled
+ if replicating data from the publisher triggers non-transient errors
+ such as referential integrity or permissions errors. The default is
+ <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>enabled</literal> (<type>boolean</type>)</term>
<listitem>
<para>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 25021e2..fcace32 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->binary = subform->subbinary;
sub->stream = subform->substream;
sub->twophasestate = subform->subtwophasestate;
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index cb2f77c..2db49db 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1258,7 +1258,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
+GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subdisableonerr, subbinary,
substream, subtwophasestate, subslotname, subsynccommit, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9427e86..23c8f3c 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -61,6 +61,7 @@
#define SUBOPT_BINARY 0x00000080
#define SUBOPT_STREAMING 0x00000100
#define SUBOPT_TWOPHASE_COMMIT 0x00000200
+#define SUBOPT_DISABLE_ON_ERR 0x00000400
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -82,6 +83,7 @@ typedef struct SubOpts
bool binary;
bool streaming;
bool twophase;
+ bool disableonerr;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -129,6 +131,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->streaming = false;
if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT))
opts->twophase = false;
+ if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
+ opts->disableonerr = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -248,6 +252,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_TWOPHASE_COMMIT;
opts->twophase = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR) &&
+ strcmp(defel->defname, "disable_on_error") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DISABLE_ON_ERR))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DISABLE_ON_ERR;
+ opts->disableonerr = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -397,7 +410,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
supported_opts = (SUBOPT_CONNECT | SUBOPT_ENABLED | SUBOPT_CREATE_SLOT |
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT);
+ SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
+ SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -465,6 +479,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled);
+ values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary);
values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming);
values[Anum_pg_subscription_subtwophasestate - 1] =
@@ -871,11 +886,19 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
{
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING);
+ SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
+ if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR))
+ {
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = BoolGetDatum(opts.disableonerr);
+ replaces[Anum_pg_subscription_subdisableonerr - 1]
+ = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_SLOT_NAME))
{
/*
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 3fb4caa..febfc4d 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -132,6 +132,7 @@ get_subscription_list(void)
sub->dbid = subform->subdbid;
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->name = pstrdup(NameStr(subform->subname));
/* We don't fill fields we are not interested in. */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 2e79302..f424421 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -136,6 +136,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/partition.h"
#include "catalog/pg_inherits.h"
@@ -334,6 +335,7 @@ static void apply_spooled_messages(TransactionId xid, XLogRecPtr lsn);
static void apply_error_callback(void *arg);
static inline void set_apply_error_context_xact(TransactionId xid, TimestampTz ts);
static inline void reset_apply_error_context_info(void);
+static bool IsSubscriptionDisablingError(ErrorData *edata);
/*
* Should this worker apply changes for given relation.
@@ -2755,6 +2757,116 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/*
+ * Errors which are transient, network protocol related, or resource exhaustion
+ * related, should not disable a subscription. These may clear up without user
+ * intervention in the subscription, schema, or data being replicated.
+ */
+static bool
+IsSubscriptionDisablingError(ErrorData *edata)
+{
+ switch (edata->sqlerrcode)
+ {
+ case ERRCODE_CONNECTION_EXCEPTION:
+ case ERRCODE_CONNECTION_DOES_NOT_EXIST:
+ case ERRCODE_CONNECTION_FAILURE:
+ case ERRCODE_SQLCLIENT_UNABLE_TO_ESTABLISH_SQLCONNECTION:
+ case ERRCODE_SQLSERVER_REJECTED_ESTABLISHMENT_OF_SQLCONNECTION:
+ case ERRCODE_TRANSACTION_RESOLUTION_UNKNOWN:
+ case ERRCODE_PROTOCOL_VIOLATION:
+ case ERRCODE_INSUFFICIENT_RESOURCES:
+ case ERRCODE_DISK_FULL:
+ case ERRCODE_OUT_OF_MEMORY:
+ case ERRCODE_TOO_MANY_CONNECTIONS:
+ case ERRCODE_CONFIGURATION_LIMIT_EXCEEDED:
+ case ERRCODE_PROGRAM_LIMIT_EXCEEDED:
+ case ERRCODE_STATEMENT_TOO_COMPLEX:
+ case ERRCODE_TOO_MANY_COLUMNS:
+ case ERRCODE_TOO_MANY_ARGUMENTS:
+ case ERRCODE_OPERATOR_INTERVENTION:
+ case ERRCODE_QUERY_CANCELED:
+ case ERRCODE_ADMIN_SHUTDOWN:
+ case ERRCODE_CRASH_SHUTDOWN:
+ case ERRCODE_CANNOT_CONNECT_NOW:
+ case ERRCODE_DATABASE_DROPPED:
+ case ERRCODE_IDLE_SESSION_TIMEOUT:
+ return false;
+ default:
+ break;
+ }
+
+ return true;
+}
+
+/*
+ * Recover from a possibly aborted transaction state and disable the current
+ * subscription.
+ */
+static ErrorData *
+DisableSubscriptionOnError(ErrorData *edata)
+{
+ Relation rel;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Oid subid;
+ Form_pg_subscription subform;
+
+ /* Disable the subscription in a fresh transaction */
+ ereport(LOG,
+ (errmsg("logical replication subscription \"%s\" will be disabled due to error: %s",
+ MySubscription->name, edata->message)));
+
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+
+ StartTransactionCommand();
+
+ /* Look up our subscription in the catalogs */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy2(SUBSCRIPTIONNAME, MyDatabaseId,
+ CStringGetDatum(MySubscription->name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("subscription \"%s\" does not exist",
+ MySubscription->name)));
+
+ subform = (Form_pg_subscription) GETSTRUCT(tup);
+ subid = subform->oid;
+ LockSharedObject(SubscriptionRelationId, subid, 0, AccessExclusiveLock);
+
+ /*
+ * We would not be here unless this subscription's disableonerr field was
+ * true when our worker began applying changes, but check whether that
+ * field has changed in the interim.
+ */
+ if (!subform->subdisableonerr)
+ ReThrowError(edata);
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Set the subscription to disabled, and note the reason. */
+ values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(false);
+ replaces[Anum_pg_subscription_subenabled - 1] = true;
+
+ /* Update the catalog */
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+ heap_freetuple(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ CommitTransactionCommand();
+
+ return edata;
+}
+
+/*
* Send a Standby Status Update message to server.
*
* 'recvpos' is the latest LSN we've received data to, force is set if we need
@@ -3339,6 +3451,8 @@ ApplyWorkerMain(Datum main_arg)
char *myslotname;
WalRcvStreamOptions options;
int server_version;
+ bool disable_subscription = false;
+ ErrorData *errdata = NULL;
/* Attach to slot */
logicalrep_worker_attach(worker_slot);
@@ -3441,7 +3555,8 @@ ApplyWorkerMain(Datum main_arg)
PG_CATCH();
{
MemoryContext ecxt = MemoryContextSwitchTo(cctx);
- ErrorData *errdata = CopyErrorData();
+
+ errdata = CopyErrorData();
/*
* Report the table sync error. There is no corresponding message
@@ -3453,11 +3568,23 @@ ApplyWorkerMain(Datum main_arg)
0, /* message type */
InvalidTransactionId,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- PG_RE_THROW();
+
+ /* Decide whether or not we disable this subscription */
+ if (MySubscription->disableonerr &&
+ IsSubscriptionDisablingError(errdata))
+ disable_subscription = true;
+ else
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
+ /* If we caught an error above, disable the subscription */
+ if (disable_subscription)
+ ReThrowError(DisableSubscriptionOnError(errdata));
+
/* allocate slot name in long-lived context */
myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
@@ -3584,7 +3711,8 @@ ApplyWorkerMain(Datum main_arg)
if (apply_error_callback_arg.command != 0)
{
MemoryContext ecxt = MemoryContextSwitchTo(cctx);
- ErrorData *errdata = CopyErrorData();
+
+ errdata = CopyErrorData();
pgstat_report_subworker_error(MyLogicalRepWorker->subid,
MyLogicalRepWorker->relid,
@@ -3594,13 +3722,28 @@ ApplyWorkerMain(Datum main_arg)
apply_error_callback_arg.command,
apply_error_callback_arg.remote_xid,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- }
- PG_RE_THROW();
+ if (MySubscription->disableonerr &&
+ IsSubscriptionDisablingError(errdata))
+ disable_subscription = true;
+ else
+ {
+ /*
+ * Some work in error recovery work is done. Switch to the old
+ * memory context.
+ */
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ }
+ else
+ PG_RE_THROW(); /* No need to change the memory context */
}
PG_END_TRY();
+ if (disable_subscription)
+ ReThrowError(DisableSubscriptionOnError(errdata));
+
proc_exit(0);
}
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index ea721d9..25a9a3f 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6508,7 +6508,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false};
+ false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6542,11 +6542,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Binary"),
gettext_noop("Streaming"));
- /* Two_phase is only supported in v15 and higher */
+ /* Two_phase and disable_on_error is only supported in v15 and higher */
if (pset.sversion >= 150000)
appendPQExpBuffer(&buf,
- ", subtwophasestate AS \"%s\"\n",
- gettext_noop("Two phase commit"));
+ ", subtwophasestate AS \"%s\"\n"
+ ", subdisableonerr AS \"%s\"\n",
+ gettext_noop("Two phase commit"),
+ gettext_noop("Disable On Err"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 4f724e4..d151265 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1681,7 +1681,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit", "disable_on_error");
/* ALTER SUBSCRIPTION <name> SET PUBLICATION */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "PUBLICATION"))
{
@@ -2898,7 +2898,7 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
"enabled", "slot_name", "streaming",
- "synchronous_commit", "two_phase");
+ "synchronous_commit", "two_phase", "disable_on_error");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 2106149..9912b90 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -60,6 +60,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subdisableonerr; /* True if apply errors should disable the
+ * subscription upon error */
+
bool subbinary; /* True if the subscription wants the
* publisher to send data in binary */
@@ -99,6 +102,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool disableonerr; /* Whether errors automatically disable */
bool binary; /* Indicates if the subscription wants data in
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 9467a19..b830be4 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -2586,6 +2586,46 @@ sub wait_for_slot_catchup
return;
}
+=pot
+
+=item $node->wait_for_subscriptions($dbname, @subcriptions)
+
+Wait for the named subscriptions to catch up or to be disabled.
+
+=cut
+
+sub wait_for_subscriptions
+{
+ my ($self, $dbname, @subscriptions) = @_;
+
+ # Unique-ify the subscriptions passed by the caller
+ my %unique = map { $_ => 1 } @subscriptions;
+ my @unique = sort keys %unique;
+ my $unique_count = scalar(@unique);
+
+ # Construct a SQL list from the unique subscription names
+ my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
+ my $sublist = join(', ', map { "'$_'" } @escaped);
+
+ my $polling_sql = qq(
+ SELECT COUNT(1) = $unique_count FROM
+ (SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ LEFT JOIN pg_catalog.pg_subscription_rel sr
+ ON sr.srsubid = s.oid
+ WHERE (sr IS NULL OR sr.srsubstate IN ('s', 'r'))
+ AND s.subname IN ($sublist)
+ AND s.subenabled IS TRUE
+ UNION
+ SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ WHERE s.subname IN ($sublist)
+ AND s.subenabled IS FALSE
+ ) AS synced_or_disabled
+ );
+ return $self->poll_query_until($dbname, $polling_sql);
+}
+
=pod
=item $node->query_hash($dbname, $query, @columns)
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 15a1ac6..57f24b9 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -94,10 +94,10 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+----------------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
@@ -129,10 +129,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+----------------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -165,19 +165,19 @@ ERROR: binary requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -188,19 +188,19 @@ ERROR: streaming requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication already exists
@@ -215,10 +215,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication used more then once
@@ -233,10 +233,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -270,10 +270,10 @@ ERROR: two_phase requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | f | off | dbname=regress_doesnotexist
(1 row)
--fail - alter of two_phase option not supported.
@@ -282,10 +282,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -294,10 +294,33 @@ DROP SUBSCRIPTION regress_testsub;
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_err must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+ERROR: disable_on_error requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | t | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7faa935..2977253 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -228,6 +228,20 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_err must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+
+\dRs+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/t/027_disable_on_error.pl b/src/test/subscription/t/027_disable_on_error.pl
new file mode 100644
index 0000000..5720c21
--- /dev/null
+++ b/src/test/subscription/t/027_disable_on_error.pl
@@ -0,0 +1,190 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+
+my @schemas = qw(s1 s2);
+my ($schema, $cmd);
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+# Create identical schema, table and index on both the publisher and
+# subscriber
+#
+for $schema (@schemas)
+{
+ $cmd = qq(
+CREATE SCHEMA $schema;
+CREATE TABLE $schema.tbl (i INT);
+ALTER TABLE $schema.tbl REPLICA IDENTITY FULL;
+CREATE INDEX ${schema}_tbl_idx ON $schema.tbl(i));
+ $node_publisher->safe_psql('postgres', $cmd);
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Create non-unique data in both schemas on the publisher.
+#
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (1), (1), (1));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Create an additional unique index in schema s1 on the subscriber only. When
+# we create subscriptions, below, this should cause subscription "s1" on the
+# subscriber to fail during initial synchronization and to get automatically
+# disabled.
+#
+$cmd = qq(CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Create publications and subscriptions linking the schemas on
+# the publisher with those on the subscriber. This tests that the
+# uniqueness violations cause subscription "s1" to fail during
+# initial synchronization.
+#
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+for $schema (@schemas)
+{
+ # Create the publication for this table
+ $cmd = qq(
+CREATE PUBLICATION $schema FOR TABLE $schema.tbl);
+ $node_publisher->safe_psql('postgres', $cmd);
+
+ # Create the subscription for this table
+ $cmd = qq(
+CREATE SUBSCRIPTION $schema
+ CONNECTION '$publisher_connstr'
+ PUBLICATION $schema
+ WITH (disable_on_error = true));
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Wait for the initial subscription synchronizations to finish or fail.
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT i, COUNT(*) FROM s2.tbl GROUP BY i);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "1|3",
+ "subscription s2 replicated initial data");
+
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+#
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (2));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that the
+# problems encountered by subscription "s1" do not cause subscription "s2" to
+# get stuck.
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should still be disabled and have replicated no data
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 still disabled");
+
+# Subscription "s2" should still be enabled and have replicated all changes
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s2 replicated data");
+
+# Drop the unique index on "s1" which caused the subscription to be disabled
+#
+$cmd = qq(DROP INDEX s1.s1_tbl_unique);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Re-enable the subscription "s1"
+#
+$cmd = q(ALTER SUBSCRIPTION s1 ENABLE);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Wait for the data to replicate
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check that we have the new data in s1.tbl
+#
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s1.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s1 replicated data");
+
+# Delete the data from the subscriber only, and recreate the unique index
+#
+$cmd = q(
+DELETE FROM s1.tbl;
+CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Add more non-unique data to the publisher
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (3), (3), (3));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that
+# uniqueness violations encountered during replication cause s1 to be disabled.
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "3|7",
+ "subscription s2 replicated additional data");
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
2.2.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-16 07:59 [email protected] <[email protected]>
parent: Greg Nancarrow <[email protected]>
0 siblings, 0 replies; 74+ messages in thread
From: [email protected] @ 2021-11-16 07:59 UTC (permalink / raw)
To: 'Greg Nancarrow' <[email protected]>; +Cc: vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
Thank you for checking the patch !
On Friday, November 12, 2021 1:49 PM Greg Nancarrow <[email protected]> wrote:
> On Thu, Nov 11, 2021 at 8:20 PM [email protected]
> <[email protected]> wrote:
> Some comments on the v4 patch:
>
> (1) Patch subject
> I think the patch subject should say "disable" instead of "disabling":
> Optionally disable subscriptions on error
Fixed.
> doc/src/sgml/ref/create_subscription.sgml
> (2) spelling mistake
> + if replicating data from the publisher triggers
> + non-transicent errors
>
> non-transicent -> non-transient
Fixed.
> (I notice Vignesh also pointed this out)
>
> src/backend/replication/logical/worker.c
> (3) calling geterrcode()
> The new IsSubscriptionDisablingError() function calls geterrcode().
> According to the function comment for geterrcode(), it is only intended for use
> in error callbacks.
> Instead of calling geterrcode(), couldn't the ErrorData from PG_CATCH block be
> passed to IsSubscriptionDisablingError() instead (from which it can get the
> sqlerrcode)?
>
> (4) DisableSubscriptionOnError
> DisableSubscriptionOnError() is again calling MemoryContextSwitch() and
> CopyErrorData().
> I think the ErrorData from the PG_CATCH block could simply be passed to
> DisableSubscriptionOnError() instead of the memory-context, and the existing
> MemoryContextSwitch() and CopyErrorData() calls could be removed from it.
>
> AFAICS, applying (3) and (4) above would make the code a lot cleaner.
Fixed.
The updated patch is shared in [1].
[1] - https://www.postgresql.org/message-id/TYCPR01MB8373771371B31E1E6CC74B0AED999%40TYCPR01MB8373.jpnprd0...
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-18 05:07 Greg Nancarrow <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Greg Nancarrow @ 2021-11-18 05:07 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Nov 16, 2021 at 6:53 PM [email protected]
<[email protected]> wrote:
>
> This v5 depends on v23 skip xid in [1].
>
A minor comment:
doc/src/sgml/ref/alter_subscription.sgml
(1) disable_on_err?
+ <literal>disable_on_err</literal>.
This doc update names the new parameter as "disable_on_err" instead of
"disable_on_error".
Also "disable_on_err" appears in a couple of the test case comments.
Regards,
Greg Nancarrow
Fujitsu Australia
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-18 07:22 [email protected] <[email protected]>
parent: Greg Nancarrow <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: [email protected] @ 2021-11-18 07:22 UTC (permalink / raw)
To: 'Greg Nancarrow' <[email protected]>; +Cc: vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thursday, November 18, 2021 2:08 PM Greg Nancarrow <[email protected]> wrote:
> A minor comment:
Thanks for your comments !
> doc/src/sgml/ref/alter_subscription.sgml
> (1) disable_on_err?
>
> + <literal>disable_on_err</literal>.
>
> This doc update names the new parameter as "disable_on_err" instead of
> "disable_on_error".
> Also "disable_on_err" appears in a couple of the test case comments.
Fixed all 3 places.
At the same time, I changed one function name
from IsSubscriptionDisablingError() to IsTransientError()
so that it can express what it checks correctly.
Of course, the return value of true or false
becomes reverse by this name change, but
This would make the function more general.
Also, its comments were fixed.
This version also depends on the v23 of skip xid [1]
[1] - https://www.postgresql.org/message-id/CAD21AoA5jupM6O%3DpYsyfaxQ1aMX-en8%3DQNgpW6KfXsg7_CS0CQ%40mail...
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v6-0001-Optionally-disable-subscriptions-on-error.patch (47.9K, ../../TYCPR01MB837308EA9546476CFA0C53F0ED9B9@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v6-0001-Optionally-disable-subscriptions-on-error.patch)
download | inline diff:
From d7eef6bc66b3137ed967ad6c9a758fcba008ad6f Mon Sep 17 00:00:00 2001
From: Osumi Takamichi <[email protected]>
Date: Thu, 18 Nov 2021 06:52:08 +0000
Subject: [PATCH v6] Optionally disable subscriptions on error
Logical replication apply workers for a subscription can easily get
stuck in an infinite loop of attempting to apply a change,
triggering an error (such as a constraint violation), exiting with
an error written to the subscription worker log, and restarting.
To partially remedy the situation, adding a new
subscription_parameter named 'disable_on_error'. To be consistent
with old behavior, the parameter defaults to false. When true, the
apply worker catches errors thrown, and for errors that are deemed
not to be transient, disables the subscription in order to break the
loop. The error is still also written to the logs.
Proposed and written originally by Mark Dilger
Taken over by Osumi Takamichi, Greg Nancarrow
Reviewed by Greg Nancarrow, Vignesh C
Discussion : https://www.postgresql.org/message-id/DB35438F-9356-4841-89A0-412709EBD3AB%40enterprisedb.com
---
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 ++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 27 +++-
src/backend/replication/logical/launcher.c | 1 +
src/backend/replication/logical/worker.c | 157 +++++++++++++++++++-
src/bin/psql/describe.c | 10 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 4 +
src/test/perl/PostgreSQL/Test/Cluster.pm | 40 +++++
src/test/regress/expected/subscription.out | 119 +++++++++------
src/test/regress/sql/subscription.sql | 14 ++
src/test/subscription/t/027_disable_on_error.pl | 190 ++++++++++++++++++++++++
14 files changed, 519 insertions(+), 66 deletions(-)
create mode 100644 src/test/subscription/t/027_disable_on_error.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 0b027cc..55922df 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -201,8 +201,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
information. The parameters that can be altered
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
- <literal>binary</literal>, and
- <literal>streaming</literal>.
+ <literal>binary</literal>,<literal>streaming</literal>
+ <literal>disable_on_error</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 990a41f..0d90d52 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -142,6 +142,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</varlistentry>
<varlistentry>
+ <term><literal>disable_on_error</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription should be automatically disabled
+ if replicating data from the publisher triggers non-transient errors
+ such as referential integrity or permissions errors. The default is
+ <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>enabled</literal> (<type>boolean</type>)</term>
<listitem>
<para>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 25021e2..fcace32 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->binary = subform->subbinary;
sub->stream = subform->substream;
sub->twophasestate = subform->subtwophasestate;
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index cb2f77c..2db49db 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1258,7 +1258,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
+GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subdisableonerr, subbinary,
substream, subtwophasestate, subslotname, subsynccommit, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9427e86..23c8f3c 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -61,6 +61,7 @@
#define SUBOPT_BINARY 0x00000080
#define SUBOPT_STREAMING 0x00000100
#define SUBOPT_TWOPHASE_COMMIT 0x00000200
+#define SUBOPT_DISABLE_ON_ERR 0x00000400
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -82,6 +83,7 @@ typedef struct SubOpts
bool binary;
bool streaming;
bool twophase;
+ bool disableonerr;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -129,6 +131,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->streaming = false;
if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT))
opts->twophase = false;
+ if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
+ opts->disableonerr = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -248,6 +252,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_TWOPHASE_COMMIT;
opts->twophase = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR) &&
+ strcmp(defel->defname, "disable_on_error") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DISABLE_ON_ERR))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DISABLE_ON_ERR;
+ opts->disableonerr = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -397,7 +410,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
supported_opts = (SUBOPT_CONNECT | SUBOPT_ENABLED | SUBOPT_CREATE_SLOT |
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT);
+ SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
+ SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -465,6 +479,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled);
+ values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary);
values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming);
values[Anum_pg_subscription_subtwophasestate - 1] =
@@ -871,11 +886,19 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
{
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING);
+ SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
+ if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR))
+ {
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = BoolGetDatum(opts.disableonerr);
+ replaces[Anum_pg_subscription_subdisableonerr - 1]
+ = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_SLOT_NAME))
{
/*
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 3fb4caa..febfc4d 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -132,6 +132,7 @@ get_subscription_list(void)
sub->dbid = subform->subdbid;
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->name = pstrdup(NameStr(subform->subname));
/* We don't fill fields we are not interested in. */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 2e79302..91388cd 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -136,6 +136,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/partition.h"
#include "catalog/pg_inherits.h"
@@ -334,6 +335,7 @@ static void apply_spooled_messages(TransactionId xid, XLogRecPtr lsn);
static void apply_error_callback(void *arg);
static inline void set_apply_error_context_xact(TransactionId xid, TimestampTz ts);
static inline void reset_apply_error_context_info(void);
+static bool IsTransientError(ErrorData *edata);
/*
* Should this worker apply changes for given relation.
@@ -2755,6 +2757,116 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/*
+ * Check if the error is transient, network protocol related, or resource exhaustion
+ * related. These may clear up without user intervention in the subscription, schema,
+ * or data being replicated.
+ */
+static bool
+IsTransientError(ErrorData *edata)
+{
+ switch (edata->sqlerrcode)
+ {
+ case ERRCODE_CONNECTION_EXCEPTION:
+ case ERRCODE_CONNECTION_DOES_NOT_EXIST:
+ case ERRCODE_CONNECTION_FAILURE:
+ case ERRCODE_SQLCLIENT_UNABLE_TO_ESTABLISH_SQLCONNECTION:
+ case ERRCODE_SQLSERVER_REJECTED_ESTABLISHMENT_OF_SQLCONNECTION:
+ case ERRCODE_TRANSACTION_RESOLUTION_UNKNOWN:
+ case ERRCODE_PROTOCOL_VIOLATION:
+ case ERRCODE_INSUFFICIENT_RESOURCES:
+ case ERRCODE_DISK_FULL:
+ case ERRCODE_OUT_OF_MEMORY:
+ case ERRCODE_TOO_MANY_CONNECTIONS:
+ case ERRCODE_CONFIGURATION_LIMIT_EXCEEDED:
+ case ERRCODE_PROGRAM_LIMIT_EXCEEDED:
+ case ERRCODE_STATEMENT_TOO_COMPLEX:
+ case ERRCODE_TOO_MANY_COLUMNS:
+ case ERRCODE_TOO_MANY_ARGUMENTS:
+ case ERRCODE_OPERATOR_INTERVENTION:
+ case ERRCODE_QUERY_CANCELED:
+ case ERRCODE_ADMIN_SHUTDOWN:
+ case ERRCODE_CRASH_SHUTDOWN:
+ case ERRCODE_CANNOT_CONNECT_NOW:
+ case ERRCODE_DATABASE_DROPPED:
+ case ERRCODE_IDLE_SESSION_TIMEOUT:
+ return true;
+ default:
+ break;
+ }
+
+ return false;
+}
+
+/*
+ * Recover from a possibly aborted transaction state and disable the current
+ * subscription.
+ */
+static ErrorData *
+DisableSubscriptionOnError(ErrorData *edata)
+{
+ Relation rel;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Oid subid;
+ Form_pg_subscription subform;
+
+ /* Disable the subscription in a fresh transaction */
+ ereport(LOG,
+ (errmsg("logical replication subscription \"%s\" will be disabled due to error: %s",
+ MySubscription->name, edata->message)));
+
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+
+ StartTransactionCommand();
+
+ /* Look up our subscription in the catalogs */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy2(SUBSCRIPTIONNAME, MyDatabaseId,
+ CStringGetDatum(MySubscription->name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("subscription \"%s\" does not exist",
+ MySubscription->name)));
+
+ subform = (Form_pg_subscription) GETSTRUCT(tup);
+ subid = subform->oid;
+ LockSharedObject(SubscriptionRelationId, subid, 0, AccessExclusiveLock);
+
+ /*
+ * We would not be here unless this subscription's disableonerr field was
+ * true when our worker began applying changes, but check whether that
+ * field has changed in the interim.
+ */
+ if (!subform->subdisableonerr)
+ ReThrowError(edata);
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Set the subscription to disabled, and note the reason. */
+ values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(false);
+ replaces[Anum_pg_subscription_subenabled - 1] = true;
+
+ /* Update the catalog */
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+ heap_freetuple(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ CommitTransactionCommand();
+
+ return edata;
+}
+
+/*
* Send a Standby Status Update message to server.
*
* 'recvpos' is the latest LSN we've received data to, force is set if we need
@@ -3339,6 +3451,8 @@ ApplyWorkerMain(Datum main_arg)
char *myslotname;
WalRcvStreamOptions options;
int server_version;
+ bool disable_subscription = false;
+ ErrorData *errdata = NULL;
/* Attach to slot */
logicalrep_worker_attach(worker_slot);
@@ -3441,7 +3555,8 @@ ApplyWorkerMain(Datum main_arg)
PG_CATCH();
{
MemoryContext ecxt = MemoryContextSwitchTo(cctx);
- ErrorData *errdata = CopyErrorData();
+
+ errdata = CopyErrorData();
/*
* Report the table sync error. There is no corresponding message
@@ -3453,11 +3568,23 @@ ApplyWorkerMain(Datum main_arg)
0, /* message type */
InvalidTransactionId,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- PG_RE_THROW();
+
+ /* Decide whether or not we disable this subscription */
+ if (MySubscription->disableonerr &&
+ !IsTransientError(errdata))
+ disable_subscription = true;
+ else
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
+ /* If we caught an error above, disable the subscription */
+ if (disable_subscription)
+ ReThrowError(DisableSubscriptionOnError(errdata));
+
/* allocate slot name in long-lived context */
myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
@@ -3584,7 +3711,8 @@ ApplyWorkerMain(Datum main_arg)
if (apply_error_callback_arg.command != 0)
{
MemoryContext ecxt = MemoryContextSwitchTo(cctx);
- ErrorData *errdata = CopyErrorData();
+
+ errdata = CopyErrorData();
pgstat_report_subworker_error(MyLogicalRepWorker->subid,
MyLogicalRepWorker->relid,
@@ -3594,13 +3722,28 @@ ApplyWorkerMain(Datum main_arg)
apply_error_callback_arg.command,
apply_error_callback_arg.remote_xid,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- }
- PG_RE_THROW();
+ if (MySubscription->disableonerr &&
+ !IsTransientError(errdata))
+ disable_subscription = true;
+ else
+ {
+ /*
+ * Some work in error recovery work is done. Switch to the old
+ * memory context.
+ */
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ }
+ else
+ PG_RE_THROW(); /* No need to change the memory context */
}
PG_END_TRY();
+ if (disable_subscription)
+ ReThrowError(DisableSubscriptionOnError(errdata));
+
proc_exit(0);
}
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index ea721d9..25a9a3f 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6508,7 +6508,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false};
+ false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6542,11 +6542,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Binary"),
gettext_noop("Streaming"));
- /* Two_phase is only supported in v15 and higher */
+ /* Two_phase and disable_on_error is only supported in v15 and higher */
if (pset.sversion >= 150000)
appendPQExpBuffer(&buf,
- ", subtwophasestate AS \"%s\"\n",
- gettext_noop("Two phase commit"));
+ ", subtwophasestate AS \"%s\"\n"
+ ", subdisableonerr AS \"%s\"\n",
+ gettext_noop("Two phase commit"),
+ gettext_noop("Disable On Err"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 4f724e4..d151265 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1681,7 +1681,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit", "disable_on_error");
/* ALTER SUBSCRIPTION <name> SET PUBLICATION */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "PUBLICATION"))
{
@@ -2898,7 +2898,7 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
"enabled", "slot_name", "streaming",
- "synchronous_commit", "two_phase");
+ "synchronous_commit", "two_phase", "disable_on_error");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 2106149..9912b90 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -60,6 +60,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subdisableonerr; /* True if apply errors should disable the
+ * subscription upon error */
+
bool subbinary; /* True if the subscription wants the
* publisher to send data in binary */
@@ -99,6 +102,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool disableonerr; /* Whether errors automatically disable */
bool binary; /* Indicates if the subscription wants data in
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 9467a19..b830be4 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -2586,6 +2586,46 @@ sub wait_for_slot_catchup
return;
}
+=pot
+
+=item $node->wait_for_subscriptions($dbname, @subcriptions)
+
+Wait for the named subscriptions to catch up or to be disabled.
+
+=cut
+
+sub wait_for_subscriptions
+{
+ my ($self, $dbname, @subscriptions) = @_;
+
+ # Unique-ify the subscriptions passed by the caller
+ my %unique = map { $_ => 1 } @subscriptions;
+ my @unique = sort keys %unique;
+ my $unique_count = scalar(@unique);
+
+ # Construct a SQL list from the unique subscription names
+ my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
+ my $sublist = join(', ', map { "'$_'" } @escaped);
+
+ my $polling_sql = qq(
+ SELECT COUNT(1) = $unique_count FROM
+ (SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ LEFT JOIN pg_catalog.pg_subscription_rel sr
+ ON sr.srsubid = s.oid
+ WHERE (sr IS NULL OR sr.srsubstate IN ('s', 'r'))
+ AND s.subname IN ($sublist)
+ AND s.subenabled IS TRUE
+ UNION
+ SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ WHERE s.subname IN ($sublist)
+ AND s.subenabled IS FALSE
+ ) AS synced_or_disabled
+ );
+ return $self->poll_query_until($dbname, $polling_sql);
+}
+
=pod
=item $node->query_hash($dbname, $query, @columns)
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 15a1ac6..97ae35d 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -94,10 +94,10 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+----------------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
@@ -129,10 +129,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+----------------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -165,19 +165,19 @@ ERROR: binary requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -188,19 +188,19 @@ ERROR: streaming requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication already exists
@@ -215,10 +215,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication used more then once
@@ -233,10 +233,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -270,10 +270,10 @@ ERROR: two_phase requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | f | off | dbname=regress_doesnotexist
(1 row)
--fail - alter of two_phase option not supported.
@@ -282,10 +282,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -294,10 +294,33 @@ DROP SUBSCRIPTION regress_testsub;
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+ERROR: disable_on_error requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | t | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7faa935..ee4a39b 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -228,6 +228,20 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+
+\dRs+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/t/027_disable_on_error.pl b/src/test/subscription/t/027_disable_on_error.pl
new file mode 100644
index 0000000..5720c21
--- /dev/null
+++ b/src/test/subscription/t/027_disable_on_error.pl
@@ -0,0 +1,190 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+
+my @schemas = qw(s1 s2);
+my ($schema, $cmd);
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+# Create identical schema, table and index on both the publisher and
+# subscriber
+#
+for $schema (@schemas)
+{
+ $cmd = qq(
+CREATE SCHEMA $schema;
+CREATE TABLE $schema.tbl (i INT);
+ALTER TABLE $schema.tbl REPLICA IDENTITY FULL;
+CREATE INDEX ${schema}_tbl_idx ON $schema.tbl(i));
+ $node_publisher->safe_psql('postgres', $cmd);
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Create non-unique data in both schemas on the publisher.
+#
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (1), (1), (1));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Create an additional unique index in schema s1 on the subscriber only. When
+# we create subscriptions, below, this should cause subscription "s1" on the
+# subscriber to fail during initial synchronization and to get automatically
+# disabled.
+#
+$cmd = qq(CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Create publications and subscriptions linking the schemas on
+# the publisher with those on the subscriber. This tests that the
+# uniqueness violations cause subscription "s1" to fail during
+# initial synchronization.
+#
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+for $schema (@schemas)
+{
+ # Create the publication for this table
+ $cmd = qq(
+CREATE PUBLICATION $schema FOR TABLE $schema.tbl);
+ $node_publisher->safe_psql('postgres', $cmd);
+
+ # Create the subscription for this table
+ $cmd = qq(
+CREATE SUBSCRIPTION $schema
+ CONNECTION '$publisher_connstr'
+ PUBLICATION $schema
+ WITH (disable_on_error = true));
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Wait for the initial subscription synchronizations to finish or fail.
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT i, COUNT(*) FROM s2.tbl GROUP BY i);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "1|3",
+ "subscription s2 replicated initial data");
+
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+#
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (2));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that the
+# problems encountered by subscription "s1" do not cause subscription "s2" to
+# get stuck.
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should still be disabled and have replicated no data
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 still disabled");
+
+# Subscription "s2" should still be enabled and have replicated all changes
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s2 replicated data");
+
+# Drop the unique index on "s1" which caused the subscription to be disabled
+#
+$cmd = qq(DROP INDEX s1.s1_tbl_unique);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Re-enable the subscription "s1"
+#
+$cmd = q(ALTER SUBSCRIPTION s1 ENABLE);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Wait for the data to replicate
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check that we have the new data in s1.tbl
+#
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s1.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s1 replicated data");
+
+# Delete the data from the subscriber only, and recreate the unique index
+#
+$cmd = q(
+DELETE FROM s1.tbl;
+CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Add more non-unique data to the publisher
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (3), (3), (3));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that
+# uniqueness violations encountered during replication cause s1 to be disabled.
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "3|7",
+ "subscription s2 replicated additional data");
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
2.2.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-22 06:52 vignesh C <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: vignesh C @ 2021-11-22 06:52 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Nov 18, 2021 at 12:52 PM [email protected]
<[email protected]> wrote:
>
> On Thursday, November 18, 2021 2:08 PM Greg Nancarrow <[email protected]> wrote:
> > A minor comment:
> Thanks for your comments !
>
> > doc/src/sgml/ref/alter_subscription.sgml
> > (1) disable_on_err?
> >
> > + <literal>disable_on_err</literal>.
> >
> > This doc update names the new parameter as "disable_on_err" instead of
> > "disable_on_error".
> > Also "disable_on_err" appears in a couple of the test case comments.
> Fixed all 3 places.
>
> At the same time, I changed one function name
> from IsSubscriptionDisablingError() to IsTransientError()
> so that it can express what it checks correctly.
> Of course, the return value of true or false
> becomes reverse by this name change, but
> This would make the function more general.
> Also, its comments were fixed.
>
> This version also depends on the v23 of skip xid [1]
Few comments:
1) Changes to handle pg_dump are missing. It should be done in
dumpSubscription and getSubscriptions
2) "And" is missing
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -201,8 +201,8 @@ ALTER SUBSCRIPTION <replaceable
class="parameter">name</replaceable> RENAME TO <
information. The parameters that can be altered
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
- <literal>binary</literal>, and
- <literal>streaming</literal>.
+ <literal>binary</literal>,<literal>streaming</literal>
+ <literal>disable_on_error</literal>.
Should be:
- <literal>binary</literal>, and
- <literal>streaming</literal>.
+ <literal>binary</literal>,<literal>streaming</literal>, and
+ <literal>disable_on_error</literal>.
3) Should we change this :
+ Specifies whether the subscription should be automatically disabled
+ if replicating data from the publisher triggers non-transient errors
+ such as referential integrity or permissions errors. The default is
+ <literal>false</literal>.
to:
+ Specifies whether the subscription should be automatically disabled
+ while replicating data from the publisher triggers
non-transient errors
+ such as referential integrity, permissions errors, etc. The
default is
+ <literal>false</literal>.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-26 14:36 [email protected] <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: [email protected] @ 2021-11-26 14:36 UTC (permalink / raw)
To: 'vignesh C' <[email protected]>; +Cc: Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Monday, November 22, 2021 3:53 PM vignesh C <[email protected]> wrote:
> Few comments:
Thank you so much for your review !
> 1) Changes to handle pg_dump are missing. It should be done in
> dumpSubscription and getSubscriptions
Fixed.
> 2) "And" is missing
> --- a/doc/src/sgml/ref/alter_subscription.sgml
> +++ b/doc/src/sgml/ref/alter_subscription.sgml
> @@ -201,8 +201,8 @@ ALTER SUBSCRIPTION <replaceable
> class="parameter">name</replaceable> RENAME TO <
> information. The parameters that can be altered
> are <literal>slot_name</literal>,
> <literal>synchronous_commit</literal>,
> - <literal>binary</literal>, and
> - <literal>streaming</literal>.
> + <literal>binary</literal>,<literal>streaming</literal>
> + <literal>disable_on_error</literal>.
> Should be:
> - <literal>binary</literal>, and
> - <literal>streaming</literal>.
> + <literal>binary</literal>,<literal>streaming</literal>, and
> + <literal>disable_on_error</literal>.
Fixed.
> 3) Should we change this :
> + Specifies whether the subscription should be automatically
> disabled
> + if replicating data from the publisher triggers non-transient errors
> + such as referential integrity or permissions errors. The default is
> + <literal>false</literal>.
> to:
> + Specifies whether the subscription should be automatically
> disabled
> + while replicating data from the publisher triggers
> non-transient errors
> + such as referential integrity, permissions errors, etc. The
> default is
> + <literal>false</literal>.
I preferred the previous description. The option
"disable_on_error" works with even one error.
If we use "while", the nuance would be like
we keep disabling a subscription more than once.
This situation happens only when user makes
the subscription enable without resolving the non-transient error,
which sounds a bit unnatural. So, I wanna keep the previous description.
If you are not satisfied with this, kindly let me know.
This v7 uses v26 of skip xid patch [1]
[1] - https://www.postgresql.org/message-id/CAD21AoDNe_O%2BCPucd_jQPu3gGGaCLNP%2BJ_kSPNecTdAM8HFPww%40mail...
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v7-0001-Optionally-disable-subscriptions-on-error.patch (50.7K, ../../TYCPR01MB837398F781D4B21A94B4EDE0ED639@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v7-0001-Optionally-disable-subscriptions-on-error.patch)
download | inline diff:
From 6ee7fb7d2ead53295b7e7011b5287973f422d741 Mon Sep 17 00:00:00 2001
From: Takamichi Osumi <[email protected]>
Date: Fri, 26 Nov 2021 13:47:25 +0000
Subject: [PATCH v7] Optionally disable subscriptions on error
Logical replication apply workers for a subscription can easily get
stuck in an infinite loop of attempting to apply a change,
triggering an error (such as a constraint violation), exiting with
an error written to the subscription worker log, and restarting.
To partially remedy the situation, adding a new
subscription_parameter named 'disable_on_error'. To be consistent
with old behavior, the parameter defaults to false. When true, the
apply worker catches errors thrown, and for errors that are deemed
not to be transient, disables the subscription in order to break the
loop. The error is still also written to the logs.
Proposed and written originally by Mark Dilger
Taken over by Osumi Takamichi, Greg Nancarrow
Reviewed by Greg Nancarrow, Vignesh C
Discussion : https://www.postgresql.org/message-id/DB35438F-9356-4841-89A0-412709EBD3AB%40enterprisedb.com
---
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 ++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 27 +++-
src/backend/replication/logical/launcher.c | 1 +
src/backend/replication/logical/worker.c | 157 +++++++++++++++++++-
src/bin/pg_dump/pg_dump.c | 17 ++-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 10 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 4 +
src/test/perl/PostgreSQL/Test/Cluster.pm | 40 +++++
src/test/regress/expected/subscription.out | 119 +++++++++------
src/test/regress/sql/subscription.sql | 14 ++
src/test/subscription/t/027_disable_on_error.pl | 190 ++++++++++++++++++++++++
16 files changed, 535 insertions(+), 68 deletions(-)
create mode 100644 src/test/subscription/t/027_disable_on_error.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 0b027cc..3109ee9 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -201,8 +201,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
information. The parameters that can be altered
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
- <literal>binary</literal>, and
- <literal>streaming</literal>.
+ <literal>binary</literal>,<literal>streaming</literal>, and
+ <literal>disable_on_error</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 990a41f..0d90d52 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -142,6 +142,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</varlistentry>
<varlistentry>
+ <term><literal>disable_on_error</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription should be automatically disabled
+ if replicating data from the publisher triggers non-transient errors
+ such as referential integrity or permissions errors. The default is
+ <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>enabled</literal> (<type>boolean</type>)</term>
<listitem>
<para>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 25021e2..fcace32 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->binary = subform->subbinary;
sub->stream = subform->substream;
sub->twophasestate = subform->subtwophasestate;
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 61b515c..4e9b124 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1258,7 +1258,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
+GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subdisableonerr, subbinary,
substream, subtwophasestate, subslotname, subsynccommit, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9427e86..23c8f3c 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -61,6 +61,7 @@
#define SUBOPT_BINARY 0x00000080
#define SUBOPT_STREAMING 0x00000100
#define SUBOPT_TWOPHASE_COMMIT 0x00000200
+#define SUBOPT_DISABLE_ON_ERR 0x00000400
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -82,6 +83,7 @@ typedef struct SubOpts
bool binary;
bool streaming;
bool twophase;
+ bool disableonerr;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -129,6 +131,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->streaming = false;
if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT))
opts->twophase = false;
+ if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
+ opts->disableonerr = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -248,6 +252,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_TWOPHASE_COMMIT;
opts->twophase = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR) &&
+ strcmp(defel->defname, "disable_on_error") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DISABLE_ON_ERR))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DISABLE_ON_ERR;
+ opts->disableonerr = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -397,7 +410,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
supported_opts = (SUBOPT_CONNECT | SUBOPT_ENABLED | SUBOPT_CREATE_SLOT |
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT);
+ SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
+ SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -465,6 +479,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled);
+ values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary);
values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming);
values[Anum_pg_subscription_subtwophasestate - 1] =
@@ -871,11 +886,19 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
{
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING);
+ SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
+ if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR))
+ {
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = BoolGetDatum(opts.disableonerr);
+ replaces[Anum_pg_subscription_subdisableonerr - 1]
+ = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_SLOT_NAME))
{
/*
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 3fb4caa..febfc4d 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -132,6 +132,7 @@ get_subscription_list(void)
sub->dbid = subform->subdbid;
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->name = pstrdup(NameStr(subform->subname));
/* We don't fill fields we are not interested in. */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 2e79302..91388cd 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -136,6 +136,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/partition.h"
#include "catalog/pg_inherits.h"
@@ -334,6 +335,7 @@ static void apply_spooled_messages(TransactionId xid, XLogRecPtr lsn);
static void apply_error_callback(void *arg);
static inline void set_apply_error_context_xact(TransactionId xid, TimestampTz ts);
static inline void reset_apply_error_context_info(void);
+static bool IsTransientError(ErrorData *edata);
/*
* Should this worker apply changes for given relation.
@@ -2755,6 +2757,116 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/*
+ * Check if the error is transient, network protocol related, or resource exhaustion
+ * related. These may clear up without user intervention in the subscription, schema,
+ * or data being replicated.
+ */
+static bool
+IsTransientError(ErrorData *edata)
+{
+ switch (edata->sqlerrcode)
+ {
+ case ERRCODE_CONNECTION_EXCEPTION:
+ case ERRCODE_CONNECTION_DOES_NOT_EXIST:
+ case ERRCODE_CONNECTION_FAILURE:
+ case ERRCODE_SQLCLIENT_UNABLE_TO_ESTABLISH_SQLCONNECTION:
+ case ERRCODE_SQLSERVER_REJECTED_ESTABLISHMENT_OF_SQLCONNECTION:
+ case ERRCODE_TRANSACTION_RESOLUTION_UNKNOWN:
+ case ERRCODE_PROTOCOL_VIOLATION:
+ case ERRCODE_INSUFFICIENT_RESOURCES:
+ case ERRCODE_DISK_FULL:
+ case ERRCODE_OUT_OF_MEMORY:
+ case ERRCODE_TOO_MANY_CONNECTIONS:
+ case ERRCODE_CONFIGURATION_LIMIT_EXCEEDED:
+ case ERRCODE_PROGRAM_LIMIT_EXCEEDED:
+ case ERRCODE_STATEMENT_TOO_COMPLEX:
+ case ERRCODE_TOO_MANY_COLUMNS:
+ case ERRCODE_TOO_MANY_ARGUMENTS:
+ case ERRCODE_OPERATOR_INTERVENTION:
+ case ERRCODE_QUERY_CANCELED:
+ case ERRCODE_ADMIN_SHUTDOWN:
+ case ERRCODE_CRASH_SHUTDOWN:
+ case ERRCODE_CANNOT_CONNECT_NOW:
+ case ERRCODE_DATABASE_DROPPED:
+ case ERRCODE_IDLE_SESSION_TIMEOUT:
+ return true;
+ default:
+ break;
+ }
+
+ return false;
+}
+
+/*
+ * Recover from a possibly aborted transaction state and disable the current
+ * subscription.
+ */
+static ErrorData *
+DisableSubscriptionOnError(ErrorData *edata)
+{
+ Relation rel;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Oid subid;
+ Form_pg_subscription subform;
+
+ /* Disable the subscription in a fresh transaction */
+ ereport(LOG,
+ (errmsg("logical replication subscription \"%s\" will be disabled due to error: %s",
+ MySubscription->name, edata->message)));
+
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+
+ StartTransactionCommand();
+
+ /* Look up our subscription in the catalogs */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy2(SUBSCRIPTIONNAME, MyDatabaseId,
+ CStringGetDatum(MySubscription->name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("subscription \"%s\" does not exist",
+ MySubscription->name)));
+
+ subform = (Form_pg_subscription) GETSTRUCT(tup);
+ subid = subform->oid;
+ LockSharedObject(SubscriptionRelationId, subid, 0, AccessExclusiveLock);
+
+ /*
+ * We would not be here unless this subscription's disableonerr field was
+ * true when our worker began applying changes, but check whether that
+ * field has changed in the interim.
+ */
+ if (!subform->subdisableonerr)
+ ReThrowError(edata);
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Set the subscription to disabled, and note the reason. */
+ values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(false);
+ replaces[Anum_pg_subscription_subenabled - 1] = true;
+
+ /* Update the catalog */
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+ heap_freetuple(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ CommitTransactionCommand();
+
+ return edata;
+}
+
+/*
* Send a Standby Status Update message to server.
*
* 'recvpos' is the latest LSN we've received data to, force is set if we need
@@ -3339,6 +3451,8 @@ ApplyWorkerMain(Datum main_arg)
char *myslotname;
WalRcvStreamOptions options;
int server_version;
+ bool disable_subscription = false;
+ ErrorData *errdata = NULL;
/* Attach to slot */
logicalrep_worker_attach(worker_slot);
@@ -3441,7 +3555,8 @@ ApplyWorkerMain(Datum main_arg)
PG_CATCH();
{
MemoryContext ecxt = MemoryContextSwitchTo(cctx);
- ErrorData *errdata = CopyErrorData();
+
+ errdata = CopyErrorData();
/*
* Report the table sync error. There is no corresponding message
@@ -3453,11 +3568,23 @@ ApplyWorkerMain(Datum main_arg)
0, /* message type */
InvalidTransactionId,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- PG_RE_THROW();
+
+ /* Decide whether or not we disable this subscription */
+ if (MySubscription->disableonerr &&
+ !IsTransientError(errdata))
+ disable_subscription = true;
+ else
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
+ /* If we caught an error above, disable the subscription */
+ if (disable_subscription)
+ ReThrowError(DisableSubscriptionOnError(errdata));
+
/* allocate slot name in long-lived context */
myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
@@ -3584,7 +3711,8 @@ ApplyWorkerMain(Datum main_arg)
if (apply_error_callback_arg.command != 0)
{
MemoryContext ecxt = MemoryContextSwitchTo(cctx);
- ErrorData *errdata = CopyErrorData();
+
+ errdata = CopyErrorData();
pgstat_report_subworker_error(MyLogicalRepWorker->subid,
MyLogicalRepWorker->relid,
@@ -3594,13 +3722,28 @@ ApplyWorkerMain(Datum main_arg)
apply_error_callback_arg.command,
apply_error_callback_arg.remote_xid,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- }
- PG_RE_THROW();
+ if (MySubscription->disableonerr &&
+ !IsTransientError(errdata))
+ disable_subscription = true;
+ else
+ {
+ /*
+ * Some work in error recovery work is done. Switch to the old
+ * memory context.
+ */
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ }
+ else
+ PG_RE_THROW(); /* No need to change the memory context */
}
PG_END_TRY();
+ if (disable_subscription)
+ ReThrowError(DisableSubscriptionOnError(errdata));
+
proc_exit(0);
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 5a2094d..3a67a8d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4451,6 +4451,7 @@ getSubscriptions(Archive *fout)
int i_rolname;
int i_substream;
int i_subtwophasestate;
+ int i_subdisableonerr;
int i_subconninfo;
int i_subslotname;
int i_subsynccommit;
@@ -4499,12 +4500,18 @@ getSubscriptions(Archive *fout)
appendPQExpBufferStr(query, " false AS substream,\n");
if (fout->remoteVersion >= 150000)
- appendPQExpBufferStr(query, " s.subtwophasestate\n");
+ appendPQExpBufferStr(query, " s.subtwophasestate,\n");
else
appendPQExpBuffer(query,
- " '%c' AS subtwophasestate\n",
+ " '%c' AS subtwophasestate,\n",
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBuffer(query, " s.subdisableonerr\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdisableonerr\n");
+
appendPQExpBufferStr(query,
"FROM pg_subscription s\n"
"WHERE s.subdbid = (SELECT oid FROM pg_database\n"
@@ -4525,6 +4532,7 @@ getSubscriptions(Archive *fout)
i_subbinary = PQfnumber(res, "subbinary");
i_substream = PQfnumber(res, "substream");
i_subtwophasestate = PQfnumber(res, "subtwophasestate");
+ i_subdisableonerr = PQfnumber(res, "subdisableonerr");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4552,6 +4560,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_substream));
subinfo[i].subtwophasestate =
pg_strdup(PQgetvalue(res, i, i_subtwophasestate));
+ subinfo[i].subdisableonerr =
+ pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
if (strlen(subinfo[i].rolname) == 0)
pg_log_warning("owner of subscription \"%s\" appears to be invalid",
@@ -4624,6 +4634,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0)
appendPQExpBufferStr(query, ", two_phase = on");
+ if (strcmp(subinfo->subdisableonerr, "f") != 0)
+ appendPQExpBufferStr(query, ", disable_on_error = on");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index d1d8608..0512111 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -655,6 +655,7 @@ typedef struct _SubscriptionInfo
char *subbinary;
char *substream;
char *subtwophasestate;
+ char *subdisableonerr;
char *subsynccommit;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index ea721d9..25a9a3f 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6508,7 +6508,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false};
+ false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6542,11 +6542,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Binary"),
gettext_noop("Streaming"));
- /* Two_phase is only supported in v15 and higher */
+ /* Two_phase and disable_on_error is only supported in v15 and higher */
if (pset.sversion >= 150000)
appendPQExpBuffer(&buf,
- ", subtwophasestate AS \"%s\"\n",
- gettext_noop("Two phase commit"));
+ ", subtwophasestate AS \"%s\"\n"
+ ", subdisableonerr AS \"%s\"\n",
+ gettext_noop("Two phase commit"),
+ gettext_noop("Disable On Err"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index fa2e195..1061c11 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1681,7 +1681,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit", "disable_on_error");
/* ALTER SUBSCRIPTION <name> SET PUBLICATION */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "PUBLICATION"))
{
@@ -2929,7 +2929,7 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
"enabled", "slot_name", "streaming",
- "synchronous_commit", "two_phase");
+ "synchronous_commit", "two_phase", "disable_on_error");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 2106149..9912b90 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -60,6 +60,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subdisableonerr; /* True if apply errors should disable the
+ * subscription upon error */
+
bool subbinary; /* True if the subscription wants the
* publisher to send data in binary */
@@ -99,6 +102,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool disableonerr; /* Whether errors automatically disable */
bool binary; /* Indicates if the subscription wants data in
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 9467a19..b830be4 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -2586,6 +2586,46 @@ sub wait_for_slot_catchup
return;
}
+=pot
+
+=item $node->wait_for_subscriptions($dbname, @subcriptions)
+
+Wait for the named subscriptions to catch up or to be disabled.
+
+=cut
+
+sub wait_for_subscriptions
+{
+ my ($self, $dbname, @subscriptions) = @_;
+
+ # Unique-ify the subscriptions passed by the caller
+ my %unique = map { $_ => 1 } @subscriptions;
+ my @unique = sort keys %unique;
+ my $unique_count = scalar(@unique);
+
+ # Construct a SQL list from the unique subscription names
+ my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
+ my $sublist = join(', ', map { "'$_'" } @escaped);
+
+ my $polling_sql = qq(
+ SELECT COUNT(1) = $unique_count FROM
+ (SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ LEFT JOIN pg_catalog.pg_subscription_rel sr
+ ON sr.srsubid = s.oid
+ WHERE (sr IS NULL OR sr.srsubstate IN ('s', 'r'))
+ AND s.subname IN ($sublist)
+ AND s.subenabled IS TRUE
+ UNION
+ SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ WHERE s.subname IN ($sublist)
+ AND s.subenabled IS FALSE
+ ) AS synced_or_disabled
+ );
+ return $self->poll_query_until($dbname, $polling_sql);
+}
+
=pod
=item $node->query_hash($dbname, $query, @columns)
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 15a1ac6..97ae35d 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -94,10 +94,10 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+----------------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
@@ -129,10 +129,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+----------------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -165,19 +165,19 @@ ERROR: binary requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -188,19 +188,19 @@ ERROR: streaming requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication already exists
@@ -215,10 +215,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication used more then once
@@ -233,10 +233,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -270,10 +270,10 @@ ERROR: two_phase requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | f | off | dbname=regress_doesnotexist
(1 row)
--fail - alter of two_phase option not supported.
@@ -282,10 +282,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -294,10 +294,33 @@ DROP SUBSCRIPTION regress_testsub;
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+ERROR: disable_on_error requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | t | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7faa935..ee4a39b 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -228,6 +228,20 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+
+\dRs+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/t/027_disable_on_error.pl b/src/test/subscription/t/027_disable_on_error.pl
new file mode 100644
index 0000000..5720c21
--- /dev/null
+++ b/src/test/subscription/t/027_disable_on_error.pl
@@ -0,0 +1,190 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+
+my @schemas = qw(s1 s2);
+my ($schema, $cmd);
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+# Create identical schema, table and index on both the publisher and
+# subscriber
+#
+for $schema (@schemas)
+{
+ $cmd = qq(
+CREATE SCHEMA $schema;
+CREATE TABLE $schema.tbl (i INT);
+ALTER TABLE $schema.tbl REPLICA IDENTITY FULL;
+CREATE INDEX ${schema}_tbl_idx ON $schema.tbl(i));
+ $node_publisher->safe_psql('postgres', $cmd);
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Create non-unique data in both schemas on the publisher.
+#
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (1), (1), (1));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Create an additional unique index in schema s1 on the subscriber only. When
+# we create subscriptions, below, this should cause subscription "s1" on the
+# subscriber to fail during initial synchronization and to get automatically
+# disabled.
+#
+$cmd = qq(CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Create publications and subscriptions linking the schemas on
+# the publisher with those on the subscriber. This tests that the
+# uniqueness violations cause subscription "s1" to fail during
+# initial synchronization.
+#
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+for $schema (@schemas)
+{
+ # Create the publication for this table
+ $cmd = qq(
+CREATE PUBLICATION $schema FOR TABLE $schema.tbl);
+ $node_publisher->safe_psql('postgres', $cmd);
+
+ # Create the subscription for this table
+ $cmd = qq(
+CREATE SUBSCRIPTION $schema
+ CONNECTION '$publisher_connstr'
+ PUBLICATION $schema
+ WITH (disable_on_error = true));
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Wait for the initial subscription synchronizations to finish or fail.
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT i, COUNT(*) FROM s2.tbl GROUP BY i);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "1|3",
+ "subscription s2 replicated initial data");
+
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+#
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (2));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that the
+# problems encountered by subscription "s1" do not cause subscription "s2" to
+# get stuck.
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should still be disabled and have replicated no data
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 still disabled");
+
+# Subscription "s2" should still be enabled and have replicated all changes
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s2 replicated data");
+
+# Drop the unique index on "s1" which caused the subscription to be disabled
+#
+$cmd = qq(DROP INDEX s1.s1_tbl_unique);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Re-enable the subscription "s1"
+#
+$cmd = q(ALTER SUBSCRIPTION s1 ENABLE);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Wait for the data to replicate
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check that we have the new data in s1.tbl
+#
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s1.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s1 replicated data");
+
+# Delete the data from the subscriber only, and recreate the unique index
+#
+$cmd = q(
+DELETE FROM s1.tbl;
+CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Add more non-unique data to the publisher
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (3), (3), (3));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that
+# uniqueness violations encountered during replication cause s1 to be disabled.
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+#
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is ($node_subscriber->safe_psql('postgres', $cmd),
+ "3|7",
+ "subscription s2 replicated additional data");
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-29 05:37 vignesh C <[email protected]>
parent: [email protected] <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: vignesh C @ 2021-11-29 05:37 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Nov 26, 2021 at 8:06 PM [email protected]
<[email protected]> wrote:
>
> On Monday, November 22, 2021 3:53 PM vignesh C <[email protected]> wrote:
> > Few comments:
> Thank you so much for your review !
>
> > 1) Changes to handle pg_dump are missing. It should be done in
> > dumpSubscription and getSubscriptions
> Fixed.
>
> > 2) "And" is missing
> > --- a/doc/src/sgml/ref/alter_subscription.sgml
> > +++ b/doc/src/sgml/ref/alter_subscription.sgml
> > @@ -201,8 +201,8 @@ ALTER SUBSCRIPTION <replaceable
> > class="parameter">name</replaceable> RENAME TO <
> > information. The parameters that can be altered
> > are <literal>slot_name</literal>,
> > <literal>synchronous_commit</literal>,
> > - <literal>binary</literal>, and
> > - <literal>streaming</literal>.
> > + <literal>binary</literal>,<literal>streaming</literal>
> > + <literal>disable_on_error</literal>.
> > Should be:
> > - <literal>binary</literal>, and
> > - <literal>streaming</literal>.
> > + <literal>binary</literal>,<literal>streaming</literal>, and
> > + <literal>disable_on_error</literal>.
> Fixed.
>
> > 3) Should we change this :
> > + Specifies whether the subscription should be automatically
> > disabled
> > + if replicating data from the publisher triggers non-transient errors
> > + such as referential integrity or permissions errors. The default is
> > + <literal>false</literal>.
> > to:
> > + Specifies whether the subscription should be automatically
> > disabled
> > + while replicating data from the publisher triggers
> > non-transient errors
> > + such as referential integrity, permissions errors, etc. The
> > default is
> > + <literal>false</literal>.
> I preferred the previous description. The option
> "disable_on_error" works with even one error.
> If we use "while", the nuance would be like
> we keep disabling a subscription more than once.
> This situation happens only when user makes
> the subscription enable without resolving the non-transient error,
> which sounds a bit unnatural. So, I wanna keep the previous description.
> If you are not satisfied with this, kindly let me know.
>
> This v7 uses v26 of skip xid patch [1]
Thanks for the updated patch, Few comments:
1) Since this function is used only from 027_disable_on_error and not
used by others, this can be moved to 027_disable_on_error:
+sub wait_for_subscriptions
+{
+ my ($self, $dbname, @subscriptions) = @_;
+
+ # Unique-ify the subscriptions passed by the caller
+ my %unique = map { $_ => 1 } @subscriptions;
+ my @unique = sort keys %unique;
+ my $unique_count = scalar(@unique);
+
+ # Construct a SQL list from the unique subscription names
+ my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
+ my $sublist = join(', ', map { "'$_'" } @escaped);
+
+ my $polling_sql = qq(
+ SELECT COUNT(1) = $unique_count FROM
+ (SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ LEFT JOIN pg_catalog.pg_subscription_rel sr
+ ON sr.srsubid = s.oid
+ WHERE (sr IS NULL OR sr.srsubstate IN
('s', 'r'))
+ AND s.subname IN ($sublist)
+ AND s.subenabled IS TRUE
+ UNION
+ SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ WHERE s.subname IN ($sublist)
+ AND s.subenabled IS FALSE
+ ) AS synced_or_disabled
+ );
+ return $self->poll_query_until($dbname, $polling_sql);
+}
2) The empty line after comment is not required, it can be removed
+# Create non-unique data in both schemas on the publisher.
+#
+for $schema (@schemas)
+{
3) Similarly it can be changed across the file
+# Wait for the initial subscription synchronizations to finish or fail.
+#
+$node_subscriber->wait_for_subscriptions('postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+#
+for $schema (@schemas)
4) Since subid is used only at one place, no need of subid variable,
you could replace subid with subform->oid in LockSharedObject
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Oid subid;
+ Form_pg_subscription subform;
+ subid = subform->oid;
+ LockSharedObject(SubscriptionRelationId, subid, 0, AccessExclusiveLock);
5) "permissions errors" should be "permission errors"
+ Specifies whether the subscription should be automatically disabled
+ if replicating data from the publisher triggers non-transient errors
+ such as referential integrity or permissions errors. The default is
+ <literal>false</literal>.
+ </para>
Regards,
Vignesh
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-30 04:09 Greg Nancarrow <[email protected]>
parent: [email protected] <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Greg Nancarrow @ 2021-11-30 04:09 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sat, Nov 27, 2021 at 1:36 AM [email protected]
<[email protected]> wrote:
>
> This v7 uses v26 of skip xid patch [1]
>
This patch no longer applies on the latest source.
Also, the patch is missing an update to doc/src/sgml/catalogs.sgml,
for the new "subdisableonerr" column of pg_subscription.
Regards,
Greg Nancarrow
Fujitsu Australia
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-30 12:04 [email protected] <[email protected]>
parent: Greg Nancarrow <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: [email protected] @ 2021-11-30 12:04 UTC (permalink / raw)
To: 'Greg Nancarrow' <[email protected]>; +Cc: vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tuesday, November 30, 2021 1:10 PM Greg Nancarrow <[email protected]> wrote:
> On Sat, Nov 27, 2021 at 1:36 AM [email protected]
> <[email protected]> wrote:
> >
> > This v7 uses v26 of skip xid patch [1]
> This patch no longer applies on the latest source.
> Also, the patch is missing an update to doc/src/sgml/catalogs.sgml, for the
> new "subdisableonerr" column of pg_subscription.
Thanks for your review !
Fixed the documentation accordingly. Further,
this comment invoked some more refactoring of codes
since I wrote some internal codes related to
'disable_on_error' in an inconsistent order.
I fixed this by keeping patch's codes
after that of 'two_phase' subscription option as much as possible.
I also conducted both pgindent and pgperltidy.
Now, I'll share the v8 that uses PG
whose commit id is after 8d74fc9 (pg_stat_subscription_workers).
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v8-0001-Optionally-disable-subscriptions-on-error.patch (50.9K, ../../TYCPR01MB83735AA021E0F614A3AB3221ED679@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v8-0001-Optionally-disable-subscriptions-on-error.patch)
download | inline diff:
From 168310f3f19d0d542a43eefef1e9c26990e923b7 Mon Sep 17 00:00:00 2001
From: Takamichi Osumi <[email protected]>
Date: Tue, 30 Nov 2021 11:20:06 +0000
Subject: [PATCH v8] Optionally disable subscriptions on error
Logical replication apply workers for a subscription can easily get
stuck in an infinite loop of attempting to apply a change,
triggering an error (such as a constraint violation), exiting with
an error written to the subscription worker log, and restarting.
To partially remedy the situation, adding a new
subscription_parameter named 'disable_on_error'. To be consistent
with old behavior, the parameter defaults to false. When true, the
apply worker catches errors thrown, and for errors that are deemed
not to be transient, disables the subscription in order to break the
loop. The error is still also written to the logs.
Proposed and written originally by Mark Dilger
Taken over by Osumi Takamichi, Greg Nancarrow
Reviewed by Greg Nancarrow, Vignesh C
Discussion : https://www.postgresql.org/message-id/DB35438F-9356-4841-89A0-412709EBD3AB%40enterprisedb.com
---
doc/src/sgml/catalogs.sgml | 10 ++
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 ++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 27 +++-
src/backend/replication/logical/launcher.c | 1 +
src/backend/replication/logical/worker.c | 155 +++++++++++++++++-
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 10 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 4 +
src/test/regress/expected/subscription.out | 119 ++++++++------
src/test/regress/sql/subscription.sql | 14 ++
src/test/subscription/t/027_disable_on_error.pl | 203 ++++++++++++++++++++++++
16 files changed, 516 insertions(+), 68 deletions(-)
create mode 100644 src/test/subscription/t/027_disable_on_error.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index c1d11be..0b83776 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7713,6 +7713,16 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<row>
<entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdisableonerr</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription will be disabled when subscription
+ worker detects an error
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
</para>
<para>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 0b027cc..3109ee9 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -201,8 +201,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
information. The parameters that can be altered
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
- <literal>binary</literal>, and
- <literal>streaming</literal>.
+ <literal>binary</literal>,<literal>streaming</literal>, and
+ <literal>disable_on_error</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 990a41f..471352a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -142,6 +142,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</varlistentry>
<varlistentry>
+ <term><literal>disable_on_error</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription should be automatically disabled
+ if replicating data from the publisher triggers non-transient errors
+ such as referential integrity or permission errors. The default is
+ <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>enabled</literal> (<type>boolean</type>)</term>
<listitem>
<para>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 25021e2..9b416dd 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -69,6 +69,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->binary = subform->subbinary;
sub->stream = subform->substream;
sub->twophasestate = subform->subtwophasestate;
+ sub->disableonerr = subform->subdisableonerr;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 61b515c..41f61bf 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1259,7 +1259,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
- substream, subtwophasestate, subslotname, subsynccommit, subpublications)
+ substream, subtwophasestate, subdisableonerr, subslotname, subsynccommit, subpublications)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_workers AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9427e86..5cb6ca7 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -61,6 +61,7 @@
#define SUBOPT_BINARY 0x00000080
#define SUBOPT_STREAMING 0x00000100
#define SUBOPT_TWOPHASE_COMMIT 0x00000200
+#define SUBOPT_DISABLE_ON_ERR 0x00000400
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -82,6 +83,7 @@ typedef struct SubOpts
bool binary;
bool streaming;
bool twophase;
+ bool disableonerr;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -129,6 +131,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->streaming = false;
if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT))
opts->twophase = false;
+ if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
+ opts->disableonerr = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -248,6 +252,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_TWOPHASE_COMMIT;
opts->twophase = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR) &&
+ strcmp(defel->defname, "disable_on_error") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DISABLE_ON_ERR))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DISABLE_ON_ERR;
+ opts->disableonerr = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -397,7 +410,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
supported_opts = (SUBOPT_CONNECT | SUBOPT_ENABLED | SUBOPT_CREATE_SLOT |
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT);
+ SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
+ SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -471,6 +485,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
CharGetDatum(opts.twophase ?
LOGICALREP_TWOPHASE_STATE_PENDING :
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -871,7 +886,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
{
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING);
+ SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -920,6 +935,14 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_substream - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR))
+ {
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = BoolGetDatum(opts.disableonerr);
+ replaces[Anum_pg_subscription_subdisableonerr - 1]
+ = true;
+ }
+
update_tuple = true;
break;
}
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 3fb4caa..febfc4d 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -132,6 +132,7 @@ get_subscription_list(void)
sub->dbid = subform->subdbid;
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->name = pstrdup(NameStr(subform->subname));
/* We don't fill fields we are not interested in. */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 2e79302..0caed4b 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -136,6 +136,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/partition.h"
#include "catalog/pg_inherits.h"
@@ -334,6 +335,7 @@ static void apply_spooled_messages(TransactionId xid, XLogRecPtr lsn);
static void apply_error_callback(void *arg);
static inline void set_apply_error_context_xact(TransactionId xid, TimestampTz ts);
static inline void reset_apply_error_context_info(void);
+static bool IsTransientError(ErrorData *edata);
/*
* Should this worker apply changes for given relation.
@@ -2755,6 +2757,114 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/*
+ * Check if the error is transient, network protocol related, or resource exhaustion
+ * related. These may clear up without user intervention in the subscription, schema,
+ * or data being replicated.
+ */
+static bool
+IsTransientError(ErrorData *edata)
+{
+ switch (edata->sqlerrcode)
+ {
+ case ERRCODE_CONNECTION_EXCEPTION:
+ case ERRCODE_CONNECTION_DOES_NOT_EXIST:
+ case ERRCODE_CONNECTION_FAILURE:
+ case ERRCODE_SQLCLIENT_UNABLE_TO_ESTABLISH_SQLCONNECTION:
+ case ERRCODE_SQLSERVER_REJECTED_ESTABLISHMENT_OF_SQLCONNECTION:
+ case ERRCODE_TRANSACTION_RESOLUTION_UNKNOWN:
+ case ERRCODE_PROTOCOL_VIOLATION:
+ case ERRCODE_INSUFFICIENT_RESOURCES:
+ case ERRCODE_DISK_FULL:
+ case ERRCODE_OUT_OF_MEMORY:
+ case ERRCODE_TOO_MANY_CONNECTIONS:
+ case ERRCODE_CONFIGURATION_LIMIT_EXCEEDED:
+ case ERRCODE_PROGRAM_LIMIT_EXCEEDED:
+ case ERRCODE_STATEMENT_TOO_COMPLEX:
+ case ERRCODE_TOO_MANY_COLUMNS:
+ case ERRCODE_TOO_MANY_ARGUMENTS:
+ case ERRCODE_OPERATOR_INTERVENTION:
+ case ERRCODE_QUERY_CANCELED:
+ case ERRCODE_ADMIN_SHUTDOWN:
+ case ERRCODE_CRASH_SHUTDOWN:
+ case ERRCODE_CANNOT_CONNECT_NOW:
+ case ERRCODE_DATABASE_DROPPED:
+ case ERRCODE_IDLE_SESSION_TIMEOUT:
+ return true;
+ default:
+ break;
+ }
+
+ return false;
+}
+
+/*
+ * Recover from a possibly aborted transaction state and disable the current
+ * subscription.
+ */
+static ErrorData *
+DisableSubscriptionOnError(ErrorData *edata)
+{
+ Relation rel;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Form_pg_subscription subform;
+
+ /* Disable the subscription in a fresh transaction */
+ ereport(LOG,
+ (errmsg("logical replication subscription \"%s\" will be disabled due to error: %s",
+ MySubscription->name, edata->message)));
+
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+
+ StartTransactionCommand();
+
+ /* Look up our subscription in the catalogs */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy2(SUBSCRIPTIONNAME, MyDatabaseId,
+ CStringGetDatum(MySubscription->name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("subscription \"%s\" does not exist",
+ MySubscription->name)));
+
+ subform = (Form_pg_subscription) GETSTRUCT(tup);
+ LockSharedObject(SubscriptionRelationId, subform->oid, 0, AccessExclusiveLock);
+
+ /*
+ * We would not be here unless this subscription's disableonerr field was
+ * true when our worker began applying changes, but check whether that
+ * field has changed in the interim.
+ */
+ if (!subform->subdisableonerr)
+ ReThrowError(edata);
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Set the subscription to disabled, and note the reason. */
+ values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(false);
+ replaces[Anum_pg_subscription_subenabled - 1] = true;
+
+ /* Update the catalog */
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+ heap_freetuple(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ CommitTransactionCommand();
+
+ return edata;
+}
+
+/*
* Send a Standby Status Update message to server.
*
* 'recvpos' is the latest LSN we've received data to, force is set if we need
@@ -3339,6 +3449,8 @@ ApplyWorkerMain(Datum main_arg)
char *myslotname;
WalRcvStreamOptions options;
int server_version;
+ bool disable_subscription = false;
+ ErrorData *errdata = NULL;
/* Attach to slot */
logicalrep_worker_attach(worker_slot);
@@ -3441,7 +3553,8 @@ ApplyWorkerMain(Datum main_arg)
PG_CATCH();
{
MemoryContext ecxt = MemoryContextSwitchTo(cctx);
- ErrorData *errdata = CopyErrorData();
+
+ errdata = CopyErrorData();
/*
* Report the table sync error. There is no corresponding message
@@ -3453,11 +3566,23 @@ ApplyWorkerMain(Datum main_arg)
0, /* message type */
InvalidTransactionId,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- PG_RE_THROW();
+
+ /* Decide whether or not we disable this subscription */
+ if (MySubscription->disableonerr &&
+ !IsTransientError(errdata))
+ disable_subscription = true;
+ else
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
+ /* If we caught an error above, disable the subscription */
+ if (disable_subscription)
+ ReThrowError(DisableSubscriptionOnError(errdata));
+
/* allocate slot name in long-lived context */
myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
@@ -3584,7 +3709,8 @@ ApplyWorkerMain(Datum main_arg)
if (apply_error_callback_arg.command != 0)
{
MemoryContext ecxt = MemoryContextSwitchTo(cctx);
- ErrorData *errdata = CopyErrorData();
+
+ errdata = CopyErrorData();
pgstat_report_subworker_error(MyLogicalRepWorker->subid,
MyLogicalRepWorker->relid,
@@ -3594,13 +3720,28 @@ ApplyWorkerMain(Datum main_arg)
apply_error_callback_arg.command,
apply_error_callback_arg.remote_xid,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- }
- PG_RE_THROW();
+ if (MySubscription->disableonerr &&
+ !IsTransientError(errdata))
+ disable_subscription = true;
+ else
+ {
+ /*
+ * Some work in error recovery work is done. Switch to the old
+ * memory context.
+ */
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ }
+ else
+ PG_RE_THROW(); /* No need to change the memory context */
}
PG_END_TRY();
+ if (disable_subscription)
+ ReThrowError(DisableSubscriptionOnError(errdata));
+
proc_exit(0);
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 5a2094d..3a67a8d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4451,6 +4451,7 @@ getSubscriptions(Archive *fout)
int i_rolname;
int i_substream;
int i_subtwophasestate;
+ int i_subdisableonerr;
int i_subconninfo;
int i_subslotname;
int i_subsynccommit;
@@ -4499,12 +4500,18 @@ getSubscriptions(Archive *fout)
appendPQExpBufferStr(query, " false AS substream,\n");
if (fout->remoteVersion >= 150000)
- appendPQExpBufferStr(query, " s.subtwophasestate\n");
+ appendPQExpBufferStr(query, " s.subtwophasestate,\n");
else
appendPQExpBuffer(query,
- " '%c' AS subtwophasestate\n",
+ " '%c' AS subtwophasestate,\n",
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBuffer(query, " s.subdisableonerr\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdisableonerr\n");
+
appendPQExpBufferStr(query,
"FROM pg_subscription s\n"
"WHERE s.subdbid = (SELECT oid FROM pg_database\n"
@@ -4525,6 +4532,7 @@ getSubscriptions(Archive *fout)
i_subbinary = PQfnumber(res, "subbinary");
i_substream = PQfnumber(res, "substream");
i_subtwophasestate = PQfnumber(res, "subtwophasestate");
+ i_subdisableonerr = PQfnumber(res, "subdisableonerr");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4552,6 +4560,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_substream));
subinfo[i].subtwophasestate =
pg_strdup(PQgetvalue(res, i, i_subtwophasestate));
+ subinfo[i].subdisableonerr =
+ pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
if (strlen(subinfo[i].rolname) == 0)
pg_log_warning("owner of subscription \"%s\" appears to be invalid",
@@ -4624,6 +4634,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0)
appendPQExpBufferStr(query, ", two_phase = on");
+ if (strcmp(subinfo->subdisableonerr, "f") != 0)
+ appendPQExpBufferStr(query, ", disable_on_error = on");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index d1d8608..0512111 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -655,6 +655,7 @@ typedef struct _SubscriptionInfo
char *subbinary;
char *substream;
char *subtwophasestate;
+ char *subdisableonerr;
char *subsynccommit;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index ea721d9..25a9a3f 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6508,7 +6508,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false};
+ false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6542,11 +6542,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Binary"),
gettext_noop("Streaming"));
- /* Two_phase is only supported in v15 and higher */
+ /* Two_phase and disable_on_error is only supported in v15 and higher */
if (pset.sversion >= 150000)
appendPQExpBuffer(&buf,
- ", subtwophasestate AS \"%s\"\n",
- gettext_noop("Two phase commit"));
+ ", subtwophasestate AS \"%s\"\n"
+ ", subdisableonerr AS \"%s\"\n",
+ gettext_noop("Two phase commit"),
+ gettext_noop("Disable On Err"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 630026d..46ed814 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1681,7 +1681,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit", "disable_on_error");
/* ALTER SUBSCRIPTION <name> SET PUBLICATION */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "PUBLICATION"))
{
@@ -2939,7 +2939,7 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
"enabled", "slot_name", "streaming",
- "synchronous_commit", "two_phase");
+ "synchronous_commit", "two_phase", "disable_on_error");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 2106149..284a90d 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -67,6 +67,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
char subtwophasestate; /* Stream two-phase transactions */
+ bool subdisableonerr; /* True if apply errors should disable the
+ * subscription upon error */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -103,6 +106,7 @@ typedef struct Subscription
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
char twophasestate; /* Allow streaming two-phase transactions */
+ bool disableonerr; /* Whether errors automatically disable */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 15a1ac6..97ae35d 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -94,10 +94,10 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+----------------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
@@ -129,10 +129,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+----------------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -165,19 +165,19 @@ ERROR: binary requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -188,19 +188,19 @@ ERROR: streaming requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication already exists
@@ -215,10 +215,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication used more then once
@@ -233,10 +233,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -270,10 +270,10 @@ ERROR: two_phase requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | f | off | dbname=regress_doesnotexist
(1 row)
--fail - alter of two_phase option not supported.
@@ -282,10 +282,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -294,10 +294,33 @@ DROP SUBSCRIPTION regress_testsub;
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+ERROR: disable_on_error requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Err | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+----------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | t | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7faa935..ee4a39b 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -228,6 +228,20 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+
+\dRs+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/t/027_disable_on_error.pl b/src/test/subscription/t/027_disable_on_error.pl
new file mode 100644
index 0000000..1104a50
--- /dev/null
+++ b/src/test/subscription/t/027_disable_on_error.pl
@@ -0,0 +1,203 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+
+# Wait for the named subscriptions to catch up or to be disabled.
+sub wait_for_subscriptions
+{
+ my ($node_name, $dbname, @subscriptions) = @_;
+
+ # Unique-ify the subscriptions passed by the caller
+ my %unique = map { $_ => 1 } @subscriptions;
+ my @unique = sort keys %unique;
+ my $unique_count = scalar(@unique);
+
+ # Construct a SQL list from the unique subscription names
+ my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
+ my $sublist = join(', ', map { "'$_'" } @escaped);
+
+ my $polling_sql = qq(
+ SELECT COUNT(1) = $unique_count FROM
+ (SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ LEFT JOIN pg_catalog.pg_subscription_rel sr
+ ON sr.srsubid = s.oid
+ WHERE (sr IS NULL OR sr.srsubstate IN ('s', 'r'))
+ AND s.subname IN ($sublist)
+ AND s.subenabled IS TRUE
+ UNION
+ SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ WHERE s.subname IN ($sublist)
+ AND s.subenabled IS FALSE
+ ) AS synced_or_disabled
+ );
+ return $node_name->poll_query_until($dbname, $polling_sql);
+}
+
+my @schemas = qw(s1 s2);
+my ($schema, $cmd);
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+# Create identical schema, table and index on both the publisher and
+# subscriber
+for $schema (@schemas)
+{
+ $cmd = qq(
+CREATE SCHEMA $schema;
+CREATE TABLE $schema.tbl (i INT);
+ALTER TABLE $schema.tbl REPLICA IDENTITY FULL;
+CREATE INDEX ${schema}_tbl_idx ON $schema.tbl(i));
+ $node_publisher->safe_psql('postgres', $cmd);
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Create non-unique data in both schemas on the publisher.
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (1), (1), (1));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Create an additional unique index in schema s1 on the subscriber only. When
+# we create subscriptions, below, this should cause subscription "s1" on the
+# subscriber to fail during initial synchronization and to get automatically
+# disabled.
+$cmd = qq(CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Create publications and subscriptions linking the schemas on
+# the publisher with those on the subscriber. This tests that the
+# uniqueness violations cause subscription "s1" to fail during
+# initial synchronization.
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+for $schema (@schemas)
+{
+ # Create the publication for this table
+ $cmd = qq(
+CREATE PUBLICATION $schema FOR TABLE $schema.tbl);
+ $node_publisher->safe_psql('postgres', $cmd);
+
+ # Create the subscription for this table
+ $cmd = qq(
+CREATE SUBSCRIPTION $schema
+ CONNECTION '$publisher_connstr'
+ PUBLICATION $schema
+ WITH (disable_on_error = true));
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Wait for the initial subscription synchronizations to finish or fail.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT i, COUNT(*) FROM s2.tbl GROUP BY i);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "1|3", "subscription s2 replicated initial data");
+
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (2));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that the
+# problems encountered by subscription "s1" do not cause subscription "s2" to
+# get stuck.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should still be disabled and have replicated no data
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 still disabled");
+
+# Subscription "s2" should still be enabled and have replicated all changes
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s2 replicated data");
+
+# Drop the unique index on "s1" which caused the subscription to be disabled
+$cmd = qq(DROP INDEX s1.s1_tbl_unique);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Re-enable the subscription "s1"
+$cmd = q(ALTER SUBSCRIPTION s1 ENABLE);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Wait for the data to replicate
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check that we have the new data in s1.tbl
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s1.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s1 replicated data");
+
+# Delete the data from the subscriber only, and recreate the unique index
+$cmd = q(
+DELETE FROM s1.tbl;
+CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Add more non-unique data to the publisher
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (3), (3), (3));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that
+# uniqueness violations encountered during replication cause s1 to be disabled.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "3|7", "subscription s2 replicated additional data");
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-11-30 12:13 [email protected] <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 0 replies; 74+ messages in thread
From: [email protected] @ 2021-11-30 12:13 UTC (permalink / raw)
To: 'vignesh C' <[email protected]>; +Cc: Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Monday, November 29, 2021 2:38 PM vignesh C <[email protected]>
> Thanks for the updated patch, Few comments:
Thank you for your review !
> 1) Since this function is used only from 027_disable_on_error and not used by
> others, this can be moved to 027_disable_on_error:
> +sub wait_for_subscriptions
> +{
> + my ($self, $dbname, @subscriptions) = @_;
> +
> + # Unique-ify the subscriptions passed by the caller
> + my %unique = map { $_ => 1 } @subscriptions;
> + my @unique = sort keys %unique;
> + my $unique_count = scalar(@unique);
> +
> + # Construct a SQL list from the unique subscription names
> + my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
> + my $sublist = join(', ', map { "'$_'" } @escaped);
> +
> + my $polling_sql = qq(
> + SELECT COUNT(1) = $unique_count FROM
> + (SELECT s.oid
> + FROM pg_catalog.pg_subscription s
> + LEFT JOIN pg_catalog.pg_subscription_rel
> sr
> + ON sr.srsubid = s.oid
> + WHERE (sr IS NULL OR sr.srsubstate IN
> ('s', 'r'))
> + AND s.subname IN ($sublist)
> + AND s.subenabled IS TRUE
> + UNION
> + SELECT s.oid
> + FROM pg_catalog.pg_subscription s
> + WHERE s.subname IN ($sublist)
> + AND s.subenabled IS FALSE
> + ) AS synced_or_disabled
> + );
> + return $self->poll_query_until($dbname, $polling_sql); }
Fixed.
> 2) The empty line after comment is not required, it can be removed
> +# Create non-unique data in both schemas on the publisher.
> +#
> +for $schema (@schemas)
> +{
Fixed.
> 3) Similarly it can be changed across the file
> +# Wait for the initial subscription synchronizations to finish or fail.
> +#
> +$node_subscriber->wait_for_subscriptions('postgres', @schemas)
> + or die "Timed out while waiting for subscriber to synchronize
> +data";
>
> +# Enter unique data for both schemas on the publisher. This should
> +succeed on # the publisher node, and not cause any additional problems
> +on the subscriber # side either, though disabled subscription "s1" should not
> replicate anything.
> +#
> +for $schema (@schemas)
Fixed.
> 4) Since subid is used only at one place, no need of subid variable, you could
> replace subid with subform->oid in LockSharedObject
> + Datum values[Natts_pg_subscription];
> + HeapTuple tup;
> + Oid subid;
> + Form_pg_subscription subform;
>
> + subid = subform->oid;
> + LockSharedObject(SubscriptionRelationId, subid, 0,
> + AccessExclusiveLock);
Fixed.
> 5) "permissions errors" should be "permission errors"
> + Specifies whether the subscription should be automatically
> disabled
> + if replicating data from the publisher triggers non-transient errors
> + such as referential integrity or permissions errors. The default is
> + <literal>false</literal>.
> + </para>
Fixed.
The new patch v8 is shared in [1].
[1] - https://www.postgresql.org/message-id/TYCPR01MB83735AA021E0F614A3AB3221ED679%40TYCPR01MB8373.jpnprd0...
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-01 06:01 vignesh C <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: vignesh C @ 2021-12-01 06:01 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Nov 30, 2021 at 5:34 PM [email protected]
<[email protected]> wrote:
>
> On Tuesday, November 30, 2021 1:10 PM Greg Nancarrow <[email protected]> wrote:
> > On Sat, Nov 27, 2021 at 1:36 AM [email protected]
> > <[email protected]> wrote:
> > >
> > > This v7 uses v26 of skip xid patch [1]
> > This patch no longer applies on the latest source.
> > Also, the patch is missing an update to doc/src/sgml/catalogs.sgml, for the
> > new "subdisableonerr" column of pg_subscription.
> Thanks for your review !
>
> Fixed the documentation accordingly. Further,
> this comment invoked some more refactoring of codes
> since I wrote some internal codes related to
> 'disable_on_error' in an inconsistent order.
> I fixed this by keeping patch's codes
> after that of 'two_phase' subscription option as much as possible.
>
> I also conducted both pgindent and pgperltidy.
>
> Now, I'll share the v8 that uses PG
> whose commit id is after 8d74fc9 (pg_stat_subscription_workers).
Thanks for the updated patch, few small comments:
1) This should be changed:
+ <structfield>subdisableonerr</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription will be disabled when subscription
+ worker detects an error
+ </para></entry>
+ </row>
to:
+ <structfield>subdisableonerr</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription will be disabled when subscription
+ worker detects non-transient errors
+ </para></entry>
+ </row>
2) "Disable On Err" can be changed to "Disable On Error"
+ ",
subtwophasestate AS \"%s\"\n"
+ ",
subdisableonerr AS \"%s\"\n",
+
gettext_noop("Two phase commit"),
+
gettext_noop("Disable On Err"));
3) Can add a line in the commit message saying "Bump catalog version."
as the patch involves changing the catalog.
4) This prototype is not required, since the function is called after
the function definition:
static inline void set_apply_error_context_xact(TransactionId xid,
TimestampTz ts);
static inline void reset_apply_error_context_info(void);
+static bool IsTransientError(ErrorData *edata);
5) we could use the new style here:
+ ereport(LOG,
+ (errmsg("logical replication subscription
\"%s\" will be disabled due to error: %s",
+ MySubscription->name, edata->message)));
change it to:
+ ereport(LOG,
+ errmsg("logical replication subscription
\"%s\" will be disabled due to error: %s",
+ MySubscription->name, edata->message));
Similarly it can be changed in the other ereports added.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-01 12:25 [email protected] <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: [email protected] @ 2021-12-01 12:25 UTC (permalink / raw)
To: 'vignesh C' <[email protected]>; +Cc: Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Amit Kapila <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wednesday, December 1, 2021 3:02 PM vignesh C <[email protected]> wrote:
> On Tue, Nov 30, 2021 at 5:34 PM [email protected]
> <[email protected]> wrote:
> >
> > On Tuesday, November 30, 2021 1:10 PM Greg Nancarrow
> <[email protected]> wrote:
> > > On Sat, Nov 27, 2021 at 1:36 AM [email protected]
> > > <[email protected]> wrote:
> > > >
> > > > This v7 uses v26 of skip xid patch [1]
> > > This patch no longer applies on the latest source.
> > > Also, the patch is missing an update to doc/src/sgml/catalogs.sgml,
> > > for the new "subdisableonerr" column of pg_subscription.
> > Thanks for your review !
> >
> > Fixed the documentation accordingly. Further, this comment invoked
> > some more refactoring of codes since I wrote some internal codes
> > related to 'disable_on_error' in an inconsistent order.
> > I fixed this by keeping patch's codes
> > after that of 'two_phase' subscription option as much as possible.
> >
> > I also conducted both pgindent and pgperltidy.
> >
> > Now, I'll share the v8 that uses PG
> > whose commit id is after 8d74fc9 (pg_stat_subscription_workers).
>
> Thanks for the updated patch, few small comments:
I appreciate your check.
> 1) This should be changed:
> + <structfield>subdisableonerr</structfield> <type>bool</type>
> + </para>
> + <para>
> + If true, the subscription will be disabled when subscription
> + worker detects an error
> + </para></entry>
> + </row>
>
> to:
> + <structfield>subdisableonerr</structfield> <type>bool</type>
> + </para>
> + <para>
> + If true, the subscription will be disabled when subscription
> + worker detects non-transient errors
> + </para></entry>
> + </row>
Fixed. Actually, there's no clear definition what "non-transient" means
in the documentation. So, I added some words to your suggestion,
which would give clearer understanding to users.
> 2) "Disable On Err" can be changed to "Disable On Error"
> + ",
> subtwophasestate AS \"%s\"\n"
> + ",
> subdisableonerr AS \"%s\"\n",
> +
> gettext_noop("Two phase commit"),
> +
> gettext_noop("Disable On Err"));
Fixed.
> 3) Can add a line in the commit message saying "Bump catalog version."
> as the patch involves changing the catalog.
Hmm, let me postpone this fix till the final version.
The catalog version gets easily updated by other patch commits
and including it in the middle of development can become
cause of conflicts of my patch when applied to the PG,
which is possible to make other reviewers stop reviewing.
> 4) This prototype is not required, since the function is called after the function
> definition:
> static inline void set_apply_error_context_xact(TransactionId xid,
> TimestampTz ts); static inline void reset_apply_error_context_info(void);
> +static bool IsTransientError(ErrorData *edata);
Fixed.
> 5) we could use the new style here:
> + ereport(LOG,
> + (errmsg("logical replication subscription
> \"%s\" will be disabled due to error: %s",
> + MySubscription->name,
> + edata->message)));
>
> change it to:
> + ereport(LOG,
> + errmsg("logical replication subscription
> \"%s\" will be disabled due to error: %s",
> + MySubscription->name,
> + edata->message));
>
> Similarly it can be changed in the other ereports added.
Removed the unnecessary parentheses.
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v9-0001-Optionally-disable-subscriptions-on-error.patch (50.7K, ../../TYCPR01MB8373369C42D82D8F85E4AA83ED689@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v9-0001-Optionally-disable-subscriptions-on-error.patch)
download | inline diff:
From 4788d7222f9eefb62f27ffd330455127bea4a0ca Mon Sep 17 00:00:00 2001
From: Takamichi Osumi <[email protected]>
Date: Wed, 1 Dec 2021 11:55:40 +0000
Subject: [PATCH v9] Optionally disable subscriptions on error
Logical replication apply workers for a subscription can easily get
stuck in an infinite loop of attempting to apply a change,
triggering an error (such as a constraint violation), exiting with
an error written to the subscription worker log, and restarting.
To partially remedy the situation, adding a new
subscription_parameter named 'disable_on_error'. To be consistent
with old behavior, the parameter defaults to false. When true, the
apply worker catches errors thrown, and for errors that are deemed
not to be transient, disables the subscription in order to break the
loop. The error is still also written to the logs.
Proposed and written originally by Mark Dilger
Taken over by Osumi Takamichi, Greg Nancarrow
Reviewed by Greg Nancarrow, Vignesh C
Discussion : https://www.postgresql.org/message-id/DB35438F-9356-4841-89A0-412709EBD3AB%40enterprisedb.com
---
doc/src/sgml/catalogs.sgml | 12 ++
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 ++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 27 +++-
src/backend/replication/logical/launcher.c | 1 +
src/backend/replication/logical/worker.c | 154 +++++++++++++++++-
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 10 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 4 +
src/test/regress/expected/subscription.out | 119 ++++++++------
src/test/regress/sql/subscription.sql | 14 ++
src/test/subscription/t/027_disable_on_error.pl | 203 ++++++++++++++++++++++++
16 files changed, 517 insertions(+), 68 deletions(-)
create mode 100644 src/test/subscription/t/027_disable_on_error.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index c1d11be..9041a75 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7713,6 +7713,18 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<row>
<entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdisableonerr</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription will be disabled when subscription
+ worker detects non-transient errors (e.g. duplication error)
+ that require user intervention in the subscription, schema,
+ or data
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
</para>
<para>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 0b027cc..3109ee9 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -201,8 +201,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
information. The parameters that can be altered
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
- <literal>binary</literal>, and
- <literal>streaming</literal>.
+ <literal>binary</literal>,<literal>streaming</literal>, and
+ <literal>disable_on_error</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 990a41f..471352a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -142,6 +142,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</varlistentry>
<varlistentry>
+ <term><literal>disable_on_error</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription should be automatically disabled
+ if replicating data from the publisher triggers non-transient errors
+ such as referential integrity or permission errors. The default is
+ <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>enabled</literal> (<type>boolean</type>)</term>
<listitem>
<para>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 25021e2..9b416dd 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -69,6 +69,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->binary = subform->subbinary;
sub->stream = subform->substream;
sub->twophasestate = subform->subtwophasestate;
+ sub->disableonerr = subform->subdisableonerr;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 61b515c..41f61bf 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1259,7 +1259,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
- substream, subtwophasestate, subslotname, subsynccommit, subpublications)
+ substream, subtwophasestate, subdisableonerr, subslotname, subsynccommit, subpublications)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_workers AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9427e86..5cb6ca7 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -61,6 +61,7 @@
#define SUBOPT_BINARY 0x00000080
#define SUBOPT_STREAMING 0x00000100
#define SUBOPT_TWOPHASE_COMMIT 0x00000200
+#define SUBOPT_DISABLE_ON_ERR 0x00000400
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -82,6 +83,7 @@ typedef struct SubOpts
bool binary;
bool streaming;
bool twophase;
+ bool disableonerr;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -129,6 +131,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->streaming = false;
if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT))
opts->twophase = false;
+ if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
+ opts->disableonerr = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -248,6 +252,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_TWOPHASE_COMMIT;
opts->twophase = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR) &&
+ strcmp(defel->defname, "disable_on_error") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DISABLE_ON_ERR))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DISABLE_ON_ERR;
+ opts->disableonerr = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -397,7 +410,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
supported_opts = (SUBOPT_CONNECT | SUBOPT_ENABLED | SUBOPT_CREATE_SLOT |
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT);
+ SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
+ SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -471,6 +485,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
CharGetDatum(opts.twophase ?
LOGICALREP_TWOPHASE_STATE_PENDING :
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -871,7 +886,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
{
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING);
+ SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -920,6 +935,14 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_substream - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR))
+ {
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = BoolGetDatum(opts.disableonerr);
+ replaces[Anum_pg_subscription_subdisableonerr - 1]
+ = true;
+ }
+
update_tuple = true;
break;
}
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 3fb4caa..febfc4d 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -132,6 +132,7 @@ get_subscription_list(void)
sub->dbid = subform->subdbid;
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->name = pstrdup(NameStr(subform->subname));
/* We don't fill fields we are not interested in. */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 2e79302..f3a63cd 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -136,6 +136,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/partition.h"
#include "catalog/pg_inherits.h"
@@ -2755,6 +2756,114 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/*
+ * Check if the error is transient, network protocol related, or resource exhaustion
+ * related. These may clear up without user intervention in the subscription, schema,
+ * or data being replicated.
+ */
+static bool
+IsTransientError(ErrorData *edata)
+{
+ switch (edata->sqlerrcode)
+ {
+ case ERRCODE_CONNECTION_EXCEPTION:
+ case ERRCODE_CONNECTION_DOES_NOT_EXIST:
+ case ERRCODE_CONNECTION_FAILURE:
+ case ERRCODE_SQLCLIENT_UNABLE_TO_ESTABLISH_SQLCONNECTION:
+ case ERRCODE_SQLSERVER_REJECTED_ESTABLISHMENT_OF_SQLCONNECTION:
+ case ERRCODE_TRANSACTION_RESOLUTION_UNKNOWN:
+ case ERRCODE_PROTOCOL_VIOLATION:
+ case ERRCODE_INSUFFICIENT_RESOURCES:
+ case ERRCODE_DISK_FULL:
+ case ERRCODE_OUT_OF_MEMORY:
+ case ERRCODE_TOO_MANY_CONNECTIONS:
+ case ERRCODE_CONFIGURATION_LIMIT_EXCEEDED:
+ case ERRCODE_PROGRAM_LIMIT_EXCEEDED:
+ case ERRCODE_STATEMENT_TOO_COMPLEX:
+ case ERRCODE_TOO_MANY_COLUMNS:
+ case ERRCODE_TOO_MANY_ARGUMENTS:
+ case ERRCODE_OPERATOR_INTERVENTION:
+ case ERRCODE_QUERY_CANCELED:
+ case ERRCODE_ADMIN_SHUTDOWN:
+ case ERRCODE_CRASH_SHUTDOWN:
+ case ERRCODE_CANNOT_CONNECT_NOW:
+ case ERRCODE_DATABASE_DROPPED:
+ case ERRCODE_IDLE_SESSION_TIMEOUT:
+ return true;
+ default:
+ break;
+ }
+
+ return false;
+}
+
+/*
+ * Recover from a possibly aborted transaction state and disable the current
+ * subscription.
+ */
+static ErrorData *
+DisableSubscriptionOnError(ErrorData *edata)
+{
+ Relation rel;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Form_pg_subscription subform;
+
+ /* Disable the subscription in a fresh transaction */
+ ereport(LOG,
+ errmsg("logical replication subscription \"%s\" will be disabled due to error: %s",
+ MySubscription->name, edata->message));
+
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+
+ StartTransactionCommand();
+
+ /* Look up our subscription in the catalogs */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy2(SUBSCRIPTIONNAME, MyDatabaseId,
+ CStringGetDatum(MySubscription->name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("subscription \"%s\" does not exist",
+ MySubscription->name));
+
+ subform = (Form_pg_subscription) GETSTRUCT(tup);
+ LockSharedObject(SubscriptionRelationId, subform->oid, 0, AccessExclusiveLock);
+
+ /*
+ * We would not be here unless this subscription's disableonerr field was
+ * true when our worker began applying changes, but check whether that
+ * field has changed in the interim.
+ */
+ if (!subform->subdisableonerr)
+ ReThrowError(edata);
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Set the subscription to disabled, and note the reason. */
+ values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(false);
+ replaces[Anum_pg_subscription_subenabled - 1] = true;
+
+ /* Update the catalog */
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+ heap_freetuple(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ CommitTransactionCommand();
+
+ return edata;
+}
+
+/*
* Send a Standby Status Update message to server.
*
* 'recvpos' is the latest LSN we've received data to, force is set if we need
@@ -3339,6 +3448,8 @@ ApplyWorkerMain(Datum main_arg)
char *myslotname;
WalRcvStreamOptions options;
int server_version;
+ bool disable_subscription = false;
+ ErrorData *errdata = NULL;
/* Attach to slot */
logicalrep_worker_attach(worker_slot);
@@ -3441,7 +3552,8 @@ ApplyWorkerMain(Datum main_arg)
PG_CATCH();
{
MemoryContext ecxt = MemoryContextSwitchTo(cctx);
- ErrorData *errdata = CopyErrorData();
+
+ errdata = CopyErrorData();
/*
* Report the table sync error. There is no corresponding message
@@ -3453,11 +3565,23 @@ ApplyWorkerMain(Datum main_arg)
0, /* message type */
InvalidTransactionId,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- PG_RE_THROW();
+
+ /* Decide whether or not we disable this subscription */
+ if (MySubscription->disableonerr &&
+ !IsTransientError(errdata))
+ disable_subscription = true;
+ else
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
+ /* If we caught an error above, disable the subscription */
+ if (disable_subscription)
+ ReThrowError(DisableSubscriptionOnError(errdata));
+
/* allocate slot name in long-lived context */
myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
@@ -3584,7 +3708,8 @@ ApplyWorkerMain(Datum main_arg)
if (apply_error_callback_arg.command != 0)
{
MemoryContext ecxt = MemoryContextSwitchTo(cctx);
- ErrorData *errdata = CopyErrorData();
+
+ errdata = CopyErrorData();
pgstat_report_subworker_error(MyLogicalRepWorker->subid,
MyLogicalRepWorker->relid,
@@ -3594,13 +3719,28 @@ ApplyWorkerMain(Datum main_arg)
apply_error_callback_arg.command,
apply_error_callback_arg.remote_xid,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- }
- PG_RE_THROW();
+ if (MySubscription->disableonerr &&
+ !IsTransientError(errdata))
+ disable_subscription = true;
+ else
+ {
+ /*
+ * Some work in error recovery work is done. Switch to the old
+ * memory context.
+ */
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ }
+ else
+ PG_RE_THROW(); /* No need to change the memory context */
}
PG_END_TRY();
+ if (disable_subscription)
+ ReThrowError(DisableSubscriptionOnError(errdata));
+
proc_exit(0);
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 5a2094d..3a67a8d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4451,6 +4451,7 @@ getSubscriptions(Archive *fout)
int i_rolname;
int i_substream;
int i_subtwophasestate;
+ int i_subdisableonerr;
int i_subconninfo;
int i_subslotname;
int i_subsynccommit;
@@ -4499,12 +4500,18 @@ getSubscriptions(Archive *fout)
appendPQExpBufferStr(query, " false AS substream,\n");
if (fout->remoteVersion >= 150000)
- appendPQExpBufferStr(query, " s.subtwophasestate\n");
+ appendPQExpBufferStr(query, " s.subtwophasestate,\n");
else
appendPQExpBuffer(query,
- " '%c' AS subtwophasestate\n",
+ " '%c' AS subtwophasestate,\n",
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBuffer(query, " s.subdisableonerr\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdisableonerr\n");
+
appendPQExpBufferStr(query,
"FROM pg_subscription s\n"
"WHERE s.subdbid = (SELECT oid FROM pg_database\n"
@@ -4525,6 +4532,7 @@ getSubscriptions(Archive *fout)
i_subbinary = PQfnumber(res, "subbinary");
i_substream = PQfnumber(res, "substream");
i_subtwophasestate = PQfnumber(res, "subtwophasestate");
+ i_subdisableonerr = PQfnumber(res, "subdisableonerr");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4552,6 +4560,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_substream));
subinfo[i].subtwophasestate =
pg_strdup(PQgetvalue(res, i, i_subtwophasestate));
+ subinfo[i].subdisableonerr =
+ pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
if (strlen(subinfo[i].rolname) == 0)
pg_log_warning("owner of subscription \"%s\" appears to be invalid",
@@ -4624,6 +4634,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0)
appendPQExpBufferStr(query, ", two_phase = on");
+ if (strcmp(subinfo->subdisableonerr, "f") != 0)
+ appendPQExpBufferStr(query, ", disable_on_error = on");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index d1d8608..0512111 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -655,6 +655,7 @@ typedef struct _SubscriptionInfo
char *subbinary;
char *substream;
char *subtwophasestate;
+ char *subdisableonerr;
char *subsynccommit;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index ea721d9..e2521a2 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6508,7 +6508,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false};
+ false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6542,11 +6542,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Binary"),
gettext_noop("Streaming"));
- /* Two_phase is only supported in v15 and higher */
+ /* Two_phase and disable_on_error is only supported in v15 and higher */
if (pset.sversion >= 150000)
appendPQExpBuffer(&buf,
- ", subtwophasestate AS \"%s\"\n",
- gettext_noop("Two phase commit"));
+ ", subtwophasestate AS \"%s\"\n"
+ ", subdisableonerr AS \"%s\"\n",
+ gettext_noop("Two phase commit"),
+ gettext_noop("Disable On Error"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 2f412ca..5497290 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1681,7 +1681,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit", "disable_on_error");
/* ALTER SUBSCRIPTION <name> SET PUBLICATION */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "PUBLICATION"))
{
@@ -2939,7 +2939,7 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
"enabled", "slot_name", "streaming",
- "synchronous_commit", "two_phase");
+ "synchronous_commit", "two_phase", "disable_on_error");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 2106149..284a90d 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -67,6 +67,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
char subtwophasestate; /* Stream two-phase transactions */
+ bool subdisableonerr; /* True if apply errors should disable the
+ * subscription upon error */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -103,6 +106,7 @@ typedef struct Subscription
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
char twophasestate; /* Allow streaming two-phase transactions */
+ bool disableonerr; /* Whether errors automatically disable */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 15a1ac6..db0bd19 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -94,10 +94,10 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
@@ -129,10 +129,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -165,19 +165,19 @@ ERROR: binary requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -188,19 +188,19 @@ ERROR: streaming requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication already exists
@@ -215,10 +215,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication used more then once
@@ -233,10 +233,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -270,10 +270,10 @@ ERROR: two_phase requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | f | off | dbname=regress_doesnotexist
(1 row)
--fail - alter of two_phase option not supported.
@@ -282,10 +282,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -294,10 +294,33 @@ DROP SUBSCRIPTION regress_testsub;
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+ERROR: disable_on_error requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | t | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7faa935..ee4a39b 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -228,6 +228,20 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+
+\dRs+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/t/027_disable_on_error.pl b/src/test/subscription/t/027_disable_on_error.pl
new file mode 100644
index 0000000..1104a50
--- /dev/null
+++ b/src/test/subscription/t/027_disable_on_error.pl
@@ -0,0 +1,203 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+
+# Wait for the named subscriptions to catch up or to be disabled.
+sub wait_for_subscriptions
+{
+ my ($node_name, $dbname, @subscriptions) = @_;
+
+ # Unique-ify the subscriptions passed by the caller
+ my %unique = map { $_ => 1 } @subscriptions;
+ my @unique = sort keys %unique;
+ my $unique_count = scalar(@unique);
+
+ # Construct a SQL list from the unique subscription names
+ my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
+ my $sublist = join(', ', map { "'$_'" } @escaped);
+
+ my $polling_sql = qq(
+ SELECT COUNT(1) = $unique_count FROM
+ (SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ LEFT JOIN pg_catalog.pg_subscription_rel sr
+ ON sr.srsubid = s.oid
+ WHERE (sr IS NULL OR sr.srsubstate IN ('s', 'r'))
+ AND s.subname IN ($sublist)
+ AND s.subenabled IS TRUE
+ UNION
+ SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ WHERE s.subname IN ($sublist)
+ AND s.subenabled IS FALSE
+ ) AS synced_or_disabled
+ );
+ return $node_name->poll_query_until($dbname, $polling_sql);
+}
+
+my @schemas = qw(s1 s2);
+my ($schema, $cmd);
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+# Create identical schema, table and index on both the publisher and
+# subscriber
+for $schema (@schemas)
+{
+ $cmd = qq(
+CREATE SCHEMA $schema;
+CREATE TABLE $schema.tbl (i INT);
+ALTER TABLE $schema.tbl REPLICA IDENTITY FULL;
+CREATE INDEX ${schema}_tbl_idx ON $schema.tbl(i));
+ $node_publisher->safe_psql('postgres', $cmd);
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Create non-unique data in both schemas on the publisher.
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (1), (1), (1));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Create an additional unique index in schema s1 on the subscriber only. When
+# we create subscriptions, below, this should cause subscription "s1" on the
+# subscriber to fail during initial synchronization and to get automatically
+# disabled.
+$cmd = qq(CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Create publications and subscriptions linking the schemas on
+# the publisher with those on the subscriber. This tests that the
+# uniqueness violations cause subscription "s1" to fail during
+# initial synchronization.
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+for $schema (@schemas)
+{
+ # Create the publication for this table
+ $cmd = qq(
+CREATE PUBLICATION $schema FOR TABLE $schema.tbl);
+ $node_publisher->safe_psql('postgres', $cmd);
+
+ # Create the subscription for this table
+ $cmd = qq(
+CREATE SUBSCRIPTION $schema
+ CONNECTION '$publisher_connstr'
+ PUBLICATION $schema
+ WITH (disable_on_error = true));
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Wait for the initial subscription synchronizations to finish or fail.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT i, COUNT(*) FROM s2.tbl GROUP BY i);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "1|3", "subscription s2 replicated initial data");
+
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (2));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that the
+# problems encountered by subscription "s1" do not cause subscription "s2" to
+# get stuck.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should still be disabled and have replicated no data
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 still disabled");
+
+# Subscription "s2" should still be enabled and have replicated all changes
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s2 replicated data");
+
+# Drop the unique index on "s1" which caused the subscription to be disabled
+$cmd = qq(DROP INDEX s1.s1_tbl_unique);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Re-enable the subscription "s1"
+$cmd = q(ALTER SUBSCRIPTION s1 ENABLE);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Wait for the data to replicate
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check that we have the new data in s1.tbl
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s1.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s1 replicated data");
+
+# Delete the data from the subscriber only, and recreate the unique index
+$cmd = q(
+DELETE FROM s1.tbl;
+CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Add more non-unique data to the publisher
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (3), (3), (3));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that
+# uniqueness violations encountered during replication cause s1 to be disabled.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "3|7", "subscription s2 replicated additional data");
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-01 13:16 Amit Kapila <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Amit Kapila @ 2021-12-01 13:16 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Dec 1, 2021 at 5:55 PM [email protected]
<[email protected]> wrote:
>
> On Wednesday, December 1, 2021 3:02 PM vignesh C <[email protected]> wrote:
> > On Tue, Nov 30, 2021 at 5:34 PM [email protected]
> > <[email protected]> wrote:
>
> > 3) Can add a line in the commit message saying "Bump catalog version."
> > as the patch involves changing the catalog.
> Hmm, let me postpone this fix till the final version.
> The catalog version gets easily updated by other patch commits
> and including it in the middle of development can become
> cause of conflicts of my patch when applied to the PG,
> which is possible to make other reviewers stop reviewing.
>
Vignesh seems to be suggesting just changing the commit message, not
the actual code. This is sort of a reminder to the committer to change
the catversion before pushing the patch. So that shouldn't cause any
conflicts while applying your patch.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-02 01:05 [email protected] <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: [email protected] @ 2021-12-02 01:05 UTC (permalink / raw)
To: 'Amit Kapila' <[email protected]>; +Cc: vignesh C <[email protected]>; Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wednesday, December 1, 2021 10:16 PM Amit Kapila <[email protected]> wrote:
> On Wed, Dec 1, 2021 at 5:55 PM [email protected]
> <[email protected]> wrote:
> >
> > On Wednesday, December 1, 2021 3:02 PM vignesh C
> <[email protected]> wrote:
> > > On Tue, Nov 30, 2021 at 5:34 PM [email protected]
> > > <[email protected]> wrote:
> >
> > > 3) Can add a line in the commit message saying "Bump catalog version."
> > > as the patch involves changing the catalog.
> > Hmm, let me postpone this fix till the final version.
> > The catalog version gets easily updated by other patch commits and
> > including it in the middle of development can become cause of
> > conflicts of my patch when applied to the PG, which is possible to
> > make other reviewers stop reviewing.
> >
>
> Vignesh seems to be suggesting just changing the commit message, not the
> actual code. This is sort of a reminder to the committer to change the catversion
> before pushing the patch. So that shouldn't cause any conflicts while applying
> your patch.
Ah, sorry for my misunderstanding.
Updated the patch to include the notification.
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v10-0001-Optionally-disable-subscriptions-on-error.patch (50.7K, ../../TYCPR01MB83730B37108684983C8F690DED699@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v10-0001-Optionally-disable-subscriptions-on-error.patch)
download | inline diff:
From a2d811296b5154f418339e12af7eb6f7310cd598 Mon Sep 17 00:00:00 2001
From: Takamichi Osumi <[email protected]>
Date: Thu, 2 Dec 2021 00:28:15 +0000
Subject: [PATCH v10] Optionally disable subscriptions on error
Logical replication apply workers for a subscription can easily get
stuck in an infinite loop of attempting to apply a change,
triggering an error (such as a constraint violation), exiting with
an error written to the subscription worker log, and restarting.
To partially remedy the situation, adding a new
subscription_parameter named 'disable_on_error'. To be consistent
with old behavior, the parameter defaults to false. When true, the
apply worker catches errors thrown, and for errors that are deemed
not to be transient, disables the subscription in order to break the
loop. The error is still also written to the logs.
Bump catalog version.
Proposed and written originally by Mark Dilger
Taken over by Osumi Takamichi, Greg Nancarrow
Reviewed by Greg Nancarrow, Vignesh C
Discussion : https://www.postgresql.org/message-id/DB35438F-9356-4841-89A0-412709EBD3AB%40enterprisedb.com
---
doc/src/sgml/catalogs.sgml | 12 ++
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 ++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 27 +++-
src/backend/replication/logical/launcher.c | 1 +
src/backend/replication/logical/worker.c | 154 +++++++++++++++++-
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 10 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 4 +
src/test/regress/expected/subscription.out | 119 ++++++++------
src/test/regress/sql/subscription.sql | 14 ++
src/test/subscription/t/027_disable_on_error.pl | 203 ++++++++++++++++++++++++
16 files changed, 517 insertions(+), 68 deletions(-)
create mode 100644 src/test/subscription/t/027_disable_on_error.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index c1d11be..9041a75 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7713,6 +7713,18 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<row>
<entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdisableonerr</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription will be disabled when subscription
+ worker detects non-transient errors (e.g. duplication error)
+ that require user intervention in the subscription, schema,
+ or data
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
</para>
<para>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 0b027cc..3109ee9 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -201,8 +201,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
information. The parameters that can be altered
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
- <literal>binary</literal>, and
- <literal>streaming</literal>.
+ <literal>binary</literal>,<literal>streaming</literal>, and
+ <literal>disable_on_error</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 990a41f..471352a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -142,6 +142,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</varlistentry>
<varlistentry>
+ <term><literal>disable_on_error</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription should be automatically disabled
+ if replicating data from the publisher triggers non-transient errors
+ such as referential integrity or permission errors. The default is
+ <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>enabled</literal> (<type>boolean</type>)</term>
<listitem>
<para>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 25021e2..9b416dd 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -69,6 +69,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->binary = subform->subbinary;
sub->stream = subform->substream;
sub->twophasestate = subform->subtwophasestate;
+ sub->disableonerr = subform->subdisableonerr;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 61b515c..41f61bf 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1259,7 +1259,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
- substream, subtwophasestate, subslotname, subsynccommit, subpublications)
+ substream, subtwophasestate, subdisableonerr, subslotname, subsynccommit, subpublications)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_workers AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9427e86..5cb6ca7 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -61,6 +61,7 @@
#define SUBOPT_BINARY 0x00000080
#define SUBOPT_STREAMING 0x00000100
#define SUBOPT_TWOPHASE_COMMIT 0x00000200
+#define SUBOPT_DISABLE_ON_ERR 0x00000400
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -82,6 +83,7 @@ typedef struct SubOpts
bool binary;
bool streaming;
bool twophase;
+ bool disableonerr;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -129,6 +131,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->streaming = false;
if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT))
opts->twophase = false;
+ if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
+ opts->disableonerr = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -248,6 +252,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_TWOPHASE_COMMIT;
opts->twophase = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR) &&
+ strcmp(defel->defname, "disable_on_error") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DISABLE_ON_ERR))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DISABLE_ON_ERR;
+ opts->disableonerr = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -397,7 +410,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
supported_opts = (SUBOPT_CONNECT | SUBOPT_ENABLED | SUBOPT_CREATE_SLOT |
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT);
+ SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
+ SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -471,6 +485,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
CharGetDatum(opts.twophase ?
LOGICALREP_TWOPHASE_STATE_PENDING :
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -871,7 +886,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
{
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING);
+ SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -920,6 +935,14 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_substream - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR))
+ {
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = BoolGetDatum(opts.disableonerr);
+ replaces[Anum_pg_subscription_subdisableonerr - 1]
+ = true;
+ }
+
update_tuple = true;
break;
}
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 3fb4caa..febfc4d 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -132,6 +132,7 @@ get_subscription_list(void)
sub->dbid = subform->subdbid;
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->name = pstrdup(NameStr(subform->subname));
/* We don't fill fields we are not interested in. */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 2e79302..f3a63cd 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -136,6 +136,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/partition.h"
#include "catalog/pg_inherits.h"
@@ -2755,6 +2756,114 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/*
+ * Check if the error is transient, network protocol related, or resource exhaustion
+ * related. These may clear up without user intervention in the subscription, schema,
+ * or data being replicated.
+ */
+static bool
+IsTransientError(ErrorData *edata)
+{
+ switch (edata->sqlerrcode)
+ {
+ case ERRCODE_CONNECTION_EXCEPTION:
+ case ERRCODE_CONNECTION_DOES_NOT_EXIST:
+ case ERRCODE_CONNECTION_FAILURE:
+ case ERRCODE_SQLCLIENT_UNABLE_TO_ESTABLISH_SQLCONNECTION:
+ case ERRCODE_SQLSERVER_REJECTED_ESTABLISHMENT_OF_SQLCONNECTION:
+ case ERRCODE_TRANSACTION_RESOLUTION_UNKNOWN:
+ case ERRCODE_PROTOCOL_VIOLATION:
+ case ERRCODE_INSUFFICIENT_RESOURCES:
+ case ERRCODE_DISK_FULL:
+ case ERRCODE_OUT_OF_MEMORY:
+ case ERRCODE_TOO_MANY_CONNECTIONS:
+ case ERRCODE_CONFIGURATION_LIMIT_EXCEEDED:
+ case ERRCODE_PROGRAM_LIMIT_EXCEEDED:
+ case ERRCODE_STATEMENT_TOO_COMPLEX:
+ case ERRCODE_TOO_MANY_COLUMNS:
+ case ERRCODE_TOO_MANY_ARGUMENTS:
+ case ERRCODE_OPERATOR_INTERVENTION:
+ case ERRCODE_QUERY_CANCELED:
+ case ERRCODE_ADMIN_SHUTDOWN:
+ case ERRCODE_CRASH_SHUTDOWN:
+ case ERRCODE_CANNOT_CONNECT_NOW:
+ case ERRCODE_DATABASE_DROPPED:
+ case ERRCODE_IDLE_SESSION_TIMEOUT:
+ return true;
+ default:
+ break;
+ }
+
+ return false;
+}
+
+/*
+ * Recover from a possibly aborted transaction state and disable the current
+ * subscription.
+ */
+static ErrorData *
+DisableSubscriptionOnError(ErrorData *edata)
+{
+ Relation rel;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Form_pg_subscription subform;
+
+ /* Disable the subscription in a fresh transaction */
+ ereport(LOG,
+ errmsg("logical replication subscription \"%s\" will be disabled due to error: %s",
+ MySubscription->name, edata->message));
+
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+
+ StartTransactionCommand();
+
+ /* Look up our subscription in the catalogs */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy2(SUBSCRIPTIONNAME, MyDatabaseId,
+ CStringGetDatum(MySubscription->name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("subscription \"%s\" does not exist",
+ MySubscription->name));
+
+ subform = (Form_pg_subscription) GETSTRUCT(tup);
+ LockSharedObject(SubscriptionRelationId, subform->oid, 0, AccessExclusiveLock);
+
+ /*
+ * We would not be here unless this subscription's disableonerr field was
+ * true when our worker began applying changes, but check whether that
+ * field has changed in the interim.
+ */
+ if (!subform->subdisableonerr)
+ ReThrowError(edata);
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Set the subscription to disabled, and note the reason. */
+ values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(false);
+ replaces[Anum_pg_subscription_subenabled - 1] = true;
+
+ /* Update the catalog */
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+ heap_freetuple(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ CommitTransactionCommand();
+
+ return edata;
+}
+
+/*
* Send a Standby Status Update message to server.
*
* 'recvpos' is the latest LSN we've received data to, force is set if we need
@@ -3339,6 +3448,8 @@ ApplyWorkerMain(Datum main_arg)
char *myslotname;
WalRcvStreamOptions options;
int server_version;
+ bool disable_subscription = false;
+ ErrorData *errdata = NULL;
/* Attach to slot */
logicalrep_worker_attach(worker_slot);
@@ -3441,7 +3552,8 @@ ApplyWorkerMain(Datum main_arg)
PG_CATCH();
{
MemoryContext ecxt = MemoryContextSwitchTo(cctx);
- ErrorData *errdata = CopyErrorData();
+
+ errdata = CopyErrorData();
/*
* Report the table sync error. There is no corresponding message
@@ -3453,11 +3565,23 @@ ApplyWorkerMain(Datum main_arg)
0, /* message type */
InvalidTransactionId,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- PG_RE_THROW();
+
+ /* Decide whether or not we disable this subscription */
+ if (MySubscription->disableonerr &&
+ !IsTransientError(errdata))
+ disable_subscription = true;
+ else
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
+ /* If we caught an error above, disable the subscription */
+ if (disable_subscription)
+ ReThrowError(DisableSubscriptionOnError(errdata));
+
/* allocate slot name in long-lived context */
myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
@@ -3584,7 +3708,8 @@ ApplyWorkerMain(Datum main_arg)
if (apply_error_callback_arg.command != 0)
{
MemoryContext ecxt = MemoryContextSwitchTo(cctx);
- ErrorData *errdata = CopyErrorData();
+
+ errdata = CopyErrorData();
pgstat_report_subworker_error(MyLogicalRepWorker->subid,
MyLogicalRepWorker->relid,
@@ -3594,13 +3719,28 @@ ApplyWorkerMain(Datum main_arg)
apply_error_callback_arg.command,
apply_error_callback_arg.remote_xid,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- }
- PG_RE_THROW();
+ if (MySubscription->disableonerr &&
+ !IsTransientError(errdata))
+ disable_subscription = true;
+ else
+ {
+ /*
+ * Some work in error recovery work is done. Switch to the old
+ * memory context.
+ */
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ }
+ else
+ PG_RE_THROW(); /* No need to change the memory context */
}
PG_END_TRY();
+ if (disable_subscription)
+ ReThrowError(DisableSubscriptionOnError(errdata));
+
proc_exit(0);
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 5a2094d..3a67a8d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4451,6 +4451,7 @@ getSubscriptions(Archive *fout)
int i_rolname;
int i_substream;
int i_subtwophasestate;
+ int i_subdisableonerr;
int i_subconninfo;
int i_subslotname;
int i_subsynccommit;
@@ -4499,12 +4500,18 @@ getSubscriptions(Archive *fout)
appendPQExpBufferStr(query, " false AS substream,\n");
if (fout->remoteVersion >= 150000)
- appendPQExpBufferStr(query, " s.subtwophasestate\n");
+ appendPQExpBufferStr(query, " s.subtwophasestate,\n");
else
appendPQExpBuffer(query,
- " '%c' AS subtwophasestate\n",
+ " '%c' AS subtwophasestate,\n",
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBuffer(query, " s.subdisableonerr\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdisableonerr\n");
+
appendPQExpBufferStr(query,
"FROM pg_subscription s\n"
"WHERE s.subdbid = (SELECT oid FROM pg_database\n"
@@ -4525,6 +4532,7 @@ getSubscriptions(Archive *fout)
i_subbinary = PQfnumber(res, "subbinary");
i_substream = PQfnumber(res, "substream");
i_subtwophasestate = PQfnumber(res, "subtwophasestate");
+ i_subdisableonerr = PQfnumber(res, "subdisableonerr");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4552,6 +4560,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_substream));
subinfo[i].subtwophasestate =
pg_strdup(PQgetvalue(res, i, i_subtwophasestate));
+ subinfo[i].subdisableonerr =
+ pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
if (strlen(subinfo[i].rolname) == 0)
pg_log_warning("owner of subscription \"%s\" appears to be invalid",
@@ -4624,6 +4634,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0)
appendPQExpBufferStr(query, ", two_phase = on");
+ if (strcmp(subinfo->subdisableonerr, "f") != 0)
+ appendPQExpBufferStr(query, ", disable_on_error = on");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index d1d8608..0512111 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -655,6 +655,7 @@ typedef struct _SubscriptionInfo
char *subbinary;
char *substream;
char *subtwophasestate;
+ char *subdisableonerr;
char *subsynccommit;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index ea721d9..e2521a2 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6508,7 +6508,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false};
+ false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6542,11 +6542,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Binary"),
gettext_noop("Streaming"));
- /* Two_phase is only supported in v15 and higher */
+ /* Two_phase and disable_on_error is only supported in v15 and higher */
if (pset.sversion >= 150000)
appendPQExpBuffer(&buf,
- ", subtwophasestate AS \"%s\"\n",
- gettext_noop("Two phase commit"));
+ ", subtwophasestate AS \"%s\"\n"
+ ", subdisableonerr AS \"%s\"\n",
+ gettext_noop("Two phase commit"),
+ gettext_noop("Disable On Error"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 2f412ca..5497290 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1681,7 +1681,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit", "disable_on_error");
/* ALTER SUBSCRIPTION <name> SET PUBLICATION */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "PUBLICATION"))
{
@@ -2939,7 +2939,7 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
"enabled", "slot_name", "streaming",
- "synchronous_commit", "two_phase");
+ "synchronous_commit", "two_phase", "disable_on_error");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 2106149..284a90d 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -67,6 +67,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
char subtwophasestate; /* Stream two-phase transactions */
+ bool subdisableonerr; /* True if apply errors should disable the
+ * subscription upon error */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -103,6 +106,7 @@ typedef struct Subscription
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
char twophasestate; /* Allow streaming two-phase transactions */
+ bool disableonerr; /* Whether errors automatically disable */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 15a1ac6..db0bd19 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -94,10 +94,10 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
@@ -129,10 +129,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -165,19 +165,19 @@ ERROR: binary requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -188,19 +188,19 @@ ERROR: streaming requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication already exists
@@ -215,10 +215,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication used more then once
@@ -233,10 +233,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -270,10 +270,10 @@ ERROR: two_phase requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | f | off | dbname=regress_doesnotexist
(1 row)
--fail - alter of two_phase option not supported.
@@ -282,10 +282,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -294,10 +294,33 @@ DROP SUBSCRIPTION regress_testsub;
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+ERROR: disable_on_error requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | t | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7faa935..ee4a39b 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -228,6 +228,20 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+
+\dRs+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/t/027_disable_on_error.pl b/src/test/subscription/t/027_disable_on_error.pl
new file mode 100644
index 0000000..1104a50
--- /dev/null
+++ b/src/test/subscription/t/027_disable_on_error.pl
@@ -0,0 +1,203 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+
+# Wait for the named subscriptions to catch up or to be disabled.
+sub wait_for_subscriptions
+{
+ my ($node_name, $dbname, @subscriptions) = @_;
+
+ # Unique-ify the subscriptions passed by the caller
+ my %unique = map { $_ => 1 } @subscriptions;
+ my @unique = sort keys %unique;
+ my $unique_count = scalar(@unique);
+
+ # Construct a SQL list from the unique subscription names
+ my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
+ my $sublist = join(', ', map { "'$_'" } @escaped);
+
+ my $polling_sql = qq(
+ SELECT COUNT(1) = $unique_count FROM
+ (SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ LEFT JOIN pg_catalog.pg_subscription_rel sr
+ ON sr.srsubid = s.oid
+ WHERE (sr IS NULL OR sr.srsubstate IN ('s', 'r'))
+ AND s.subname IN ($sublist)
+ AND s.subenabled IS TRUE
+ UNION
+ SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ WHERE s.subname IN ($sublist)
+ AND s.subenabled IS FALSE
+ ) AS synced_or_disabled
+ );
+ return $node_name->poll_query_until($dbname, $polling_sql);
+}
+
+my @schemas = qw(s1 s2);
+my ($schema, $cmd);
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+# Create identical schema, table and index on both the publisher and
+# subscriber
+for $schema (@schemas)
+{
+ $cmd = qq(
+CREATE SCHEMA $schema;
+CREATE TABLE $schema.tbl (i INT);
+ALTER TABLE $schema.tbl REPLICA IDENTITY FULL;
+CREATE INDEX ${schema}_tbl_idx ON $schema.tbl(i));
+ $node_publisher->safe_psql('postgres', $cmd);
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Create non-unique data in both schemas on the publisher.
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (1), (1), (1));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Create an additional unique index in schema s1 on the subscriber only. When
+# we create subscriptions, below, this should cause subscription "s1" on the
+# subscriber to fail during initial synchronization and to get automatically
+# disabled.
+$cmd = qq(CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Create publications and subscriptions linking the schemas on
+# the publisher with those on the subscriber. This tests that the
+# uniqueness violations cause subscription "s1" to fail during
+# initial synchronization.
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+for $schema (@schemas)
+{
+ # Create the publication for this table
+ $cmd = qq(
+CREATE PUBLICATION $schema FOR TABLE $schema.tbl);
+ $node_publisher->safe_psql('postgres', $cmd);
+
+ # Create the subscription for this table
+ $cmd = qq(
+CREATE SUBSCRIPTION $schema
+ CONNECTION '$publisher_connstr'
+ PUBLICATION $schema
+ WITH (disable_on_error = true));
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Wait for the initial subscription synchronizations to finish or fail.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT i, COUNT(*) FROM s2.tbl GROUP BY i);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "1|3", "subscription s2 replicated initial data");
+
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (2));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that the
+# problems encountered by subscription "s1" do not cause subscription "s2" to
+# get stuck.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should still be disabled and have replicated no data
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 still disabled");
+
+# Subscription "s2" should still be enabled and have replicated all changes
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s2 replicated data");
+
+# Drop the unique index on "s1" which caused the subscription to be disabled
+$cmd = qq(DROP INDEX s1.s1_tbl_unique);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Re-enable the subscription "s1"
+$cmd = q(ALTER SUBSCRIPTION s1 ENABLE);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Wait for the data to replicate
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check that we have the new data in s1.tbl
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s1.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s1 replicated data");
+
+# Delete the data from the subscriber only, and recreate the unique index
+$cmd = q(
+DELETE FROM s1.tbl;
+CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Add more non-unique data to the publisher
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (3), (3), (3));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that
+# uniqueness violations encountered during replication cause s1 to be disabled.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "3|7", "subscription s2 replicated additional data");
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-02 02:42 Greg Nancarrow <[email protected]>
parent: [email protected] <[email protected]>
1 sibling, 0 replies; 74+ messages in thread
From: Greg Nancarrow @ 2021-12-02 02:42 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Dec 2, 2021 at 12:05 PM [email protected]
<[email protected]> wrote:
>
> Updated the patch to include the notification.
>
For the catalogs.sgml update, I was thinking that the following
wording might sound a bit better:
+ If true, the subscription will be disabled when a subscription
+ worker detects non-transient errors (e.g. duplication error)
+ that require user intervention to resolve.
What do you think?
Regards,
Greg Nancarrow
Fujitsu Australia
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-02 04:48 Amit Kapila <[email protected]>
parent: [email protected] <[email protected]>
1 sibling, 2 replies; 74+ messages in thread
From: Amit Kapila @ 2021-12-02 04:48 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Dec 2, 2021 at 6:35 AM [email protected]
<[email protected]> wrote:
>
> On Wednesday, December 1, 2021 10:16 PM Amit Kapila <[email protected]> wrote:
> Updated the patch to include the notification.
>
The patch disables the subscription for non-transient errors. I am not
sure if we can easily make the call to decide whether any particular
error is transient or not. For example, DISK_FULL or OUT_OF_MEMORY
might not rectify itself. Why not just allow to disable the
subscription on any error? And then let the user check the error
either in view or logs and decide whether it would like to enable the
subscription or do something before it (like making space in disk, or
fixing the network).
The other problem I see with this transient error stuff is maintaining
the list of error codes that we think are transient. I think we need a
discussion for each of the error_codes we are listing now and whatever
new error_code we add in the future which doesn't seem like a good
idea.
I think the code to deal with apply worker errors and then disable the
subscription has some flaws. Say, while disabling the subscription if
it leads to another error then I think the original error won't be
reported. Can't we simply emit the error via EmitErrorReport and then
do AbortOutOfAnyTransaction, FlushErrorState, and any other memory
context clean up if required and then disable the subscription after
coming out of catch?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-02 07:40 [email protected] <[email protected]>
parent: Amit Kapila <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: [email protected] @ 2021-12-02 07:40 UTC (permalink / raw)
To: 'Amit Kapila' <[email protected]>; +Cc: vignesh C <[email protected]>; Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thursday, December 2, 2021 1:49 PM Amit Kapila <[email protected]> wrote:
> On Thu, Dec 2, 2021 at 6:35 AM [email protected]
> <[email protected]> wrote:
> >
> > On Wednesday, December 1, 2021 10:16 PM Amit Kapila
> <[email protected]> wrote:
> > Updated the patch to include the notification.
> >
> The patch disables the subscription for non-transient errors. I am not sure if we
> can easily make the call to decide whether any particular error is transient or
> not. For example, DISK_FULL or OUT_OF_MEMORY might not rectify itself.
> Why not just allow to disable the subscription on any error? And then let the
> user check the error either in view or logs and decide whether it would like to
> enable the subscription or do something before it (like making space in disk, or
> fixing the network).
Agreed. I'll treat any errors as the trigger of the feature
in the next version.
> The other problem I see with this transient error stuff is maintaining the list of
> error codes that we think are transient. I think we need a discussion for each of
> the error_codes we are listing now and whatever new error_code we add in the
> future which doesn't seem like a good idea.
This is also true. The maintenance cost of my current implementation
didn't sound cheap.
> I think the code to deal with apply worker errors and then disable the
> subscription has some flaws. Say, while disabling the subscription if it leads to
> another error then I think the original error won't be reported. Can't we simply
> emit the error via EmitErrorReport and then do AbortOutOfAnyTransaction,
> FlushErrorState, and any other memory context clean up if required and then
> disable the subscription after coming out of catch?
You are right. I'll fix related parts accordingly.
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-03 13:20 [email protected] <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: [email protected] @ 2021-12-03 13:20 UTC (permalink / raw)
To: 'Amit Kapila' <[email protected]>; +Cc: vignesh C <[email protected]>; Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
Thursday, December 2, 2021 4:41 PM I wrote:
> On Thursday, December 2, 2021 1:49 PM Amit Kapila
> <[email protected]> wrote:
> > On Thu, Dec 2, 2021 at 6:35 AM [email protected]
> > <[email protected]> wrote:
> > >
> > > On Wednesday, December 1, 2021 10:16 PM Amit Kapila
> > <[email protected]> wrote:
> > > Updated the patch to include the notification.
> > >
> > The patch disables the subscription for non-transient errors. I am not
> > sure if we can easily make the call to decide whether any particular
> > error is transient or not. For example, DISK_FULL or OUT_OF_MEMORY
> might not rectify itself.
> > Why not just allow to disable the subscription on any error? And then
> > let the user check the error either in view or logs and decide whether
> > it would like to enable the subscription or do something before it
> > (like making space in disk, or fixing the network).
> Agreed. I'll treat any errors as the trigger of the feature in the next version.
>
> > The other problem I see with this transient error stuff is maintaining
> > the list of error codes that we think are transient. I think we need a
> > discussion for each of the error_codes we are listing now and whatever
> > new error_code we add in the future which doesn't seem like a good idea.
> This is also true. The maintenance cost of my current implementation didn't
> sound cheap.
>
> > I think the code to deal with apply worker errors and then disable the
> > subscription has some flaws. Say, while disabling the subscription if
> > it leads to another error then I think the original error won't be
> > reported. Can't we simply emit the error via EmitErrorReport and then
> > do AbortOutOfAnyTransaction, FlushErrorState, and any other memory
> > context clean up if required and then disable the subscription after coming
> out of catch?
> You are right. I'll fix related parts accordingly.
Hi, I've made a new patch v11 that incorporated suggestions described above.
There are several notes to share regarding v11 modifications.
1. Modified the commit message a bit.
2. DisableSubscriptionOnError() doesn't return ErrData anymore,
since now to emit error message is done in the error recovery area
and the function purpose has become purely to run a transaction to disable
the subscription.
3. In DisableSubscriptionOnError(), v10 rethrew the error if the disable_on_error
flag became false in the interim, but v11 just closes the transaction and
finishes the function.
4. If table sync worker detects an error during synchronization
and needs to disable the subscription, the worker disables it and just exit by proc_exit.
The processing after disabling the subscription didn't look necessary to me
for disabled subscription.
5. Only when we succeed in the table synchronization, it's necessary to
allocate slot name in long-lived context, after the table synchronization in
ApplyWorkerMain(). Otherwise, we'll see junk value of syncslotname
because it is the return value of LogicalRepSyncTableStart().
6. There are 3 places for error recovery in ApplyWorkerMain().
All of those might look similar but I didn't make an united function for them.
Those are slightly different from each other and I felt
readability is reduced by grouping them into one type of function call.
7. In v11, I covered the case that apply worker failed with
apply_error_callback_arg.command == 0, as one path to disable the subscription
in order to cover all errors.
8. I changed one flag name from 'disable_subscription' to 'did_error'
in ApplyWorkerMain().
9. All chages in this version are C codes and checked by pgindent.
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v11-0001-Optionally-disable-subscriptions-on-error.patch (49.6K, ../../TYCPR01MB83732011D83A027C32A2802CED6A9@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v11-0001-Optionally-disable-subscriptions-on-error.patch)
download | inline diff:
From 4047906cfdc07f34276bc3cce84992071883001a Mon Sep 17 00:00:00 2001
From: Takamichi Osumi <[email protected]>
Date: Fri, 3 Dec 2021 12:02:55 +0000
Subject: [PATCH v11] Optionally disable subscriptions on error
Logical replication apply workers for a subscription can easily get
stuck in an infinite loop of attempting to apply a change,
triggering an error (such as a constraint violation), exiting with
an error written to the subscription worker log, and restarting.
To partially remedy the situation, adding a new
subscription_parameter named 'disable_on_error'. To be consistent
with old behavior, the parameter defaults to false. When true, both
the table sync worker and apply worker catch any errors thrown and
disable the subscription in order to break the loop. The error is
still also written to the logs.
Require to bump catalog version.
Proposed and written originally by Mark Dilger
Taken over by Osumi Takamichi, Greg Nancarrow
Reviewed by Greg Nancarrow, Vignesh C
Discussion : https://www.postgresql.org/message-id/DB35438F-9356-4841-89A0-412709EBD3AB%40enterprisedb.com
---
doc/src/sgml/catalogs.sgml | 10 ++
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 11 ++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 27 +++-
src/backend/replication/logical/launcher.c | 1 +
src/backend/replication/logical/worker.c | 148 +++++++++++++++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 10 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 4 +
src/test/regress/expected/subscription.out | 119 ++++++++------
src/test/regress/sql/subscription.sql | 14 ++
src/test/subscription/t/027_disable_on_error.pl | 203 ++++++++++++++++++++++++
16 files changed, 506 insertions(+), 70 deletions(-)
create mode 100644 src/test/subscription/t/027_disable_on_error.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index c1d11be..e879796 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7713,6 +7713,16 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<row>
<entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdisableonerr</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription will be disabled when subscription
+ worker detects any errors
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
</para>
<para>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 0b027cc..3109ee9 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -201,8 +201,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
information. The parameters that can be altered
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
- <literal>binary</literal>, and
- <literal>streaming</literal>.
+ <literal>binary</literal>,<literal>streaming</literal>, and
+ <literal>disable_on_error</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 990a41f..dacf733 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -142,6 +142,17 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</varlistentry>
<varlistentry>
+ <term><literal>disable_on_error</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription should be automatically disabled
+ if replicating data from the publisher triggers errors. The default
+ is <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>enabled</literal> (<type>boolean</type>)</term>
<listitem>
<para>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 25021e2..9b416dd 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -69,6 +69,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->binary = subform->subbinary;
sub->stream = subform->substream;
sub->twophasestate = subform->subtwophasestate;
+ sub->disableonerr = subform->subdisableonerr;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 61b515c..41f61bf 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1259,7 +1259,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
- substream, subtwophasestate, subslotname, subsynccommit, subpublications)
+ substream, subtwophasestate, subdisableonerr, subslotname, subsynccommit, subpublications)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_workers AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9427e86..5cb6ca7 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -61,6 +61,7 @@
#define SUBOPT_BINARY 0x00000080
#define SUBOPT_STREAMING 0x00000100
#define SUBOPT_TWOPHASE_COMMIT 0x00000200
+#define SUBOPT_DISABLE_ON_ERR 0x00000400
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -82,6 +83,7 @@ typedef struct SubOpts
bool binary;
bool streaming;
bool twophase;
+ bool disableonerr;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -129,6 +131,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->streaming = false;
if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT))
opts->twophase = false;
+ if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
+ opts->disableonerr = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -248,6 +252,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_TWOPHASE_COMMIT;
opts->twophase = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR) &&
+ strcmp(defel->defname, "disable_on_error") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DISABLE_ON_ERR))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DISABLE_ON_ERR;
+ opts->disableonerr = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -397,7 +410,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
supported_opts = (SUBOPT_CONNECT | SUBOPT_ENABLED | SUBOPT_CREATE_SLOT |
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT);
+ SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
+ SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -471,6 +485,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
CharGetDatum(opts.twophase ?
LOGICALREP_TWOPHASE_STATE_PENDING :
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -871,7 +886,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
{
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING);
+ SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -920,6 +935,14 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_substream - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR))
+ {
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = BoolGetDatum(opts.disableonerr);
+ replaces[Anum_pg_subscription_subdisableonerr - 1]
+ = true;
+ }
+
update_tuple = true;
break;
}
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 3fb4caa..febfc4d 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -132,6 +132,7 @@ get_subscription_list(void)
sub->dbid = subform->subdbid;
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->name = pstrdup(NameStr(subform->subname));
/* We don't fill fields we are not interested in. */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 2e79302..5e9a6c3 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -136,6 +136,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/partition.h"
#include "catalog/pg_inherits.h"
@@ -2755,6 +2756,85 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/*
+ * As a preparation for disabling the subscription, emit the error,
+ * handle the transaction and clean up the memory context of
+ * error. ErrorContext is reset by FlushErrorState.
+ */
+static void
+WorkerErrorRecovery(void)
+{
+ EmitErrorReport();
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+}
+
+/*
+ * Recover from a possibly aborted transaction state and disable the current
+ * subscription.
+ */
+static void
+DisableSubscriptionOnError(void)
+{
+ Relation rel;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Form_pg_subscription subform;
+
+ /* Disable the subscription in a fresh transaction */
+ StartTransactionCommand();
+
+ /* Look up our subscription in the catalogs */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy2(SUBSCRIPTIONNAME, MyDatabaseId,
+ CStringGetDatum(MySubscription->name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("subscription \"%s\" does not exist",
+ MySubscription->name));
+
+ subform = (Form_pg_subscription) GETSTRUCT(tup);
+ LockSharedObject(SubscriptionRelationId, subform->oid, 0, AccessExclusiveLock);
+
+ /*
+ * We would not be here unless this subscription's disableonerr field was
+ * true when our worker began applying changes, but check whether that
+ * field has changed in the interim.
+ */
+ if (!subform->subdisableonerr)
+ {
+ /*
+ * Disabling the subscription has been done already. No need of
+ * additional work.
+ */
+ table_close(rel, RowExclusiveLock);
+ CommitTransactionCommand();
+ return;
+ }
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Set the subscription to disabled, and note the reason. */
+ values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(false);
+ replaces[Anum_pg_subscription_subenabled - 1] = true;
+
+ /* Update the catalog */
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+ heap_freetuple(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ CommitTransactionCommand();
+}
+
+/*
* Send a Standby Status Update message to server.
*
* 'recvpos' is the latest LSN we've received data to, force is set if we need
@@ -3339,6 +3419,7 @@ ApplyWorkerMain(Datum main_arg)
char *myslotname;
WalRcvStreamOptions options;
int server_version;
+ bool did_error = false;
/* Attach to slot */
logicalrep_worker_attach(worker_slot);
@@ -3453,15 +3534,33 @@ ApplyWorkerMain(Datum main_arg)
0, /* message type */
InvalidTransactionId,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- PG_RE_THROW();
+
+ if (MySubscription->disableonerr)
+ {
+ did_error = true;
+ WorkerErrorRecovery();
+ }
+ else
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
- /* allocate slot name in long-lived context */
- myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
+ /* If we caught an error above, disable the subscription */
+ if (did_error)
+ {
+ DisableSubscriptionOnError();
+ proc_exit(0);
+ }
+ else
+ {
+ /* allocate slot name in long-lived context */
+ myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
- pfree(syncslotname);
+ pfree(syncslotname);
+ }
}
else
{
@@ -3580,10 +3679,11 @@ ApplyWorkerMain(Datum main_arg)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(cctx);
+
/* report the apply error */
if (apply_error_callback_arg.command != 0)
{
- MemoryContext ecxt = MemoryContextSwitchTo(cctx);
ErrorData *errdata = CopyErrorData();
pgstat_report_subworker_error(MyLogicalRepWorker->subid,
@@ -3594,13 +3694,43 @@ ApplyWorkerMain(Datum main_arg)
apply_error_callback_arg.command,
apply_error_callback_arg.remote_xid,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- }
- PG_RE_THROW();
+ if (MySubscription->disableonerr)
+ {
+ did_error = true;
+ WorkerErrorRecovery();
+ }
+ else
+ {
+ /*
+ * Some work in error recovery work is done. Switch to the old
+ * memory context and rethrow.
+ */
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ }
+ else
+ {
+ /*
+ * Don't miss any error, even when it's not reported to stats
+ * collector.
+ */
+ if (MySubscription->disableonerr)
+ {
+ did_error = true;
+ WorkerErrorRecovery();
+ }
+ else
+ /* Simply rethrow because of no recovery work */
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
+ if (did_error)
+ DisableSubscriptionOnError();
+
proc_exit(0);
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 01ffa5b..05656e3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4452,6 +4452,7 @@ getSubscriptions(Archive *fout)
int i_rolname;
int i_substream;
int i_subtwophasestate;
+ int i_subdisableonerr;
int i_subconninfo;
int i_subslotname;
int i_subsynccommit;
@@ -4500,12 +4501,18 @@ getSubscriptions(Archive *fout)
appendPQExpBufferStr(query, " false AS substream,\n");
if (fout->remoteVersion >= 150000)
- appendPQExpBufferStr(query, " s.subtwophasestate\n");
+ appendPQExpBufferStr(query, " s.subtwophasestate,\n");
else
appendPQExpBuffer(query,
- " '%c' AS subtwophasestate\n",
+ " '%c' AS subtwophasestate,\n",
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBuffer(query, " s.subdisableonerr\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdisableonerr\n");
+
appendPQExpBufferStr(query,
"FROM pg_subscription s\n"
"WHERE s.subdbid = (SELECT oid FROM pg_database\n"
@@ -4526,6 +4533,7 @@ getSubscriptions(Archive *fout)
i_subbinary = PQfnumber(res, "subbinary");
i_substream = PQfnumber(res, "substream");
i_subtwophasestate = PQfnumber(res, "subtwophasestate");
+ i_subdisableonerr = PQfnumber(res, "subdisableonerr");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4553,6 +4561,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_substream));
subinfo[i].subtwophasestate =
pg_strdup(PQgetvalue(res, i, i_subtwophasestate));
+ subinfo[i].subdisableonerr =
+ pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
if (strlen(subinfo[i].rolname) == 0)
pg_log_warning("owner of subscription \"%s\" appears to be invalid",
@@ -4625,6 +4635,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0)
appendPQExpBufferStr(query, ", two_phase = on");
+ if (strcmp(subinfo->subdisableonerr, "f") != 0)
+ appendPQExpBufferStr(query, ", disable_on_error = on");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index d1d8608..0512111 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -655,6 +655,7 @@ typedef struct _SubscriptionInfo
char *subbinary;
char *substream;
char *subtwophasestate;
+ char *subdisableonerr;
char *subsynccommit;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index ea721d9..e2521a2 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6508,7 +6508,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false};
+ false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6542,11 +6542,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Binary"),
gettext_noop("Streaming"));
- /* Two_phase is only supported in v15 and higher */
+ /* Two_phase and disable_on_error is only supported in v15 and higher */
if (pset.sversion >= 150000)
appendPQExpBuffer(&buf,
- ", subtwophasestate AS \"%s\"\n",
- gettext_noop("Two phase commit"));
+ ", subtwophasestate AS \"%s\"\n"
+ ", subdisableonerr AS \"%s\"\n",
+ gettext_noop("Two phase commit"),
+ gettext_noop("Disable On Error"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 2f412ca..5497290 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1681,7 +1681,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit", "disable_on_error");
/* ALTER SUBSCRIPTION <name> SET PUBLICATION */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "PUBLICATION"))
{
@@ -2939,7 +2939,7 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
"enabled", "slot_name", "streaming",
- "synchronous_commit", "two_phase");
+ "synchronous_commit", "two_phase", "disable_on_error");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 2106149..284a90d 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -67,6 +67,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
char subtwophasestate; /* Stream two-phase transactions */
+ bool subdisableonerr; /* True if apply errors should disable the
+ * subscription upon error */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -103,6 +106,7 @@ typedef struct Subscription
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
char twophasestate; /* Allow streaming two-phase transactions */
+ bool disableonerr; /* Whether errors automatically disable */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 15a1ac6..db0bd19 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -94,10 +94,10 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
@@ -129,10 +129,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -165,19 +165,19 @@ ERROR: binary requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -188,19 +188,19 @@ ERROR: streaming requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication already exists
@@ -215,10 +215,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication used more then once
@@ -233,10 +233,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -270,10 +270,10 @@ ERROR: two_phase requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | f | off | dbname=regress_doesnotexist
(1 row)
--fail - alter of two_phase option not supported.
@@ -282,10 +282,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -294,10 +294,33 @@ DROP SUBSCRIPTION regress_testsub;
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+ERROR: disable_on_error requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | t | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7faa935..ee4a39b 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -228,6 +228,20 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+
+\dRs+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/t/027_disable_on_error.pl b/src/test/subscription/t/027_disable_on_error.pl
new file mode 100644
index 0000000..1104a50
--- /dev/null
+++ b/src/test/subscription/t/027_disable_on_error.pl
@@ -0,0 +1,203 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+
+# Wait for the named subscriptions to catch up or to be disabled.
+sub wait_for_subscriptions
+{
+ my ($node_name, $dbname, @subscriptions) = @_;
+
+ # Unique-ify the subscriptions passed by the caller
+ my %unique = map { $_ => 1 } @subscriptions;
+ my @unique = sort keys %unique;
+ my $unique_count = scalar(@unique);
+
+ # Construct a SQL list from the unique subscription names
+ my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
+ my $sublist = join(', ', map { "'$_'" } @escaped);
+
+ my $polling_sql = qq(
+ SELECT COUNT(1) = $unique_count FROM
+ (SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ LEFT JOIN pg_catalog.pg_subscription_rel sr
+ ON sr.srsubid = s.oid
+ WHERE (sr IS NULL OR sr.srsubstate IN ('s', 'r'))
+ AND s.subname IN ($sublist)
+ AND s.subenabled IS TRUE
+ UNION
+ SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ WHERE s.subname IN ($sublist)
+ AND s.subenabled IS FALSE
+ ) AS synced_or_disabled
+ );
+ return $node_name->poll_query_until($dbname, $polling_sql);
+}
+
+my @schemas = qw(s1 s2);
+my ($schema, $cmd);
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+# Create identical schema, table and index on both the publisher and
+# subscriber
+for $schema (@schemas)
+{
+ $cmd = qq(
+CREATE SCHEMA $schema;
+CREATE TABLE $schema.tbl (i INT);
+ALTER TABLE $schema.tbl REPLICA IDENTITY FULL;
+CREATE INDEX ${schema}_tbl_idx ON $schema.tbl(i));
+ $node_publisher->safe_psql('postgres', $cmd);
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Create non-unique data in both schemas on the publisher.
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (1), (1), (1));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Create an additional unique index in schema s1 on the subscriber only. When
+# we create subscriptions, below, this should cause subscription "s1" on the
+# subscriber to fail during initial synchronization and to get automatically
+# disabled.
+$cmd = qq(CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Create publications and subscriptions linking the schemas on
+# the publisher with those on the subscriber. This tests that the
+# uniqueness violations cause subscription "s1" to fail during
+# initial synchronization.
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+for $schema (@schemas)
+{
+ # Create the publication for this table
+ $cmd = qq(
+CREATE PUBLICATION $schema FOR TABLE $schema.tbl);
+ $node_publisher->safe_psql('postgres', $cmd);
+
+ # Create the subscription for this table
+ $cmd = qq(
+CREATE SUBSCRIPTION $schema
+ CONNECTION '$publisher_connstr'
+ PUBLICATION $schema
+ WITH (disable_on_error = true));
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Wait for the initial subscription synchronizations to finish or fail.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT i, COUNT(*) FROM s2.tbl GROUP BY i);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "1|3", "subscription s2 replicated initial data");
+
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (2));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that the
+# problems encountered by subscription "s1" do not cause subscription "s2" to
+# get stuck.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should still be disabled and have replicated no data
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 still disabled");
+
+# Subscription "s2" should still be enabled and have replicated all changes
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s2 replicated data");
+
+# Drop the unique index on "s1" which caused the subscription to be disabled
+$cmd = qq(DROP INDEX s1.s1_tbl_unique);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Re-enable the subscription "s1"
+$cmd = q(ALTER SUBSCRIPTION s1 ENABLE);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Wait for the data to replicate
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check that we have the new data in s1.tbl
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s1.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s1 replicated data");
+
+# Delete the data from the subscriber only, and recreate the unique index
+$cmd = q(
+DELETE FROM s1.tbl;
+CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Add more non-unique data to the publisher
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (3), (3), (3));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that
+# uniqueness violations encountered during replication cause s1 to be disabled.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "3|7", "subscription s2 replicated additional data");
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-06 04:16 Greg Nancarrow <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Greg Nancarrow @ 2021-12-06 04:16 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sat, Dec 4, 2021 at 12:20 AM [email protected]
<[email protected]> wrote:
>
> Hi, I've made a new patch v11 that incorporated suggestions described above.
>
Some review comments for the v11 patch:
doc/src/sgml/ref/create_subscription.sgml
(1) Possible wording improvement?
BEFORE:
+ Specifies whether the subscription should be automatically disabled
+ if replicating data from the publisher triggers errors. The default
+ is <literal>false</literal>.
AFTER:
+ Specifies whether the subscription should be automatically disabled
+ if any errors are detected by subscription workers during data
+ replication from the publisher. The default is <literal>false</literal>.
src/backend/replication/logical/worker.c
(2) WorkerErrorRecovery comments
Instead of:
+ * As a preparation for disabling the subscription, emit the error,
+ * handle the transaction and clean up the memory context of
+ * error. ErrorContext is reset by FlushErrorState.
why not just say:
+ Worker error recovery processing, in preparation for disabling the
+ subscription.
And then comment the function's code lines:
e.g.
/* Emit the error */
...
/* Abort any active transaction */
...
/* Reset the ErrorContext */
...
(3) DisableSubscriptionOnError return
The "if (!subform->subdisableonerr)" block should probably first:
heap_freetuple(tup);
(regardless of the fact the only current caller will proc_exit anyway)
(4) did_error flag
I think perhaps the previously-used flag name "disable_subscription"
is better, or maybe "error_recovery_done".
Also, I think it would look better if it was set AFTER
WorkerErrorRecovery() was called.
(5) DisableSubscriptionOnError LOG message
This version of the patch removes the LOG message:
+ ereport(LOG,
+ errmsg("logical replication subscription \"%s\" will be disabled due
to error: %s",
+ MySubscription->name, edata->message));
Perhaps a similar error message could be logged prior to EmitErrorReport()?
e.g.
"logical replication subscription \"%s\" will be disabled due to an error"
Regards,
Greg Nancarrow
Fujitsu Australia
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-06 04:37 Mark Dilger <[email protected]>
parent: Amit Kapila <[email protected]>
1 sibling, 2 replies; 74+ messages in thread
From: Mark Dilger @ 2021-12-06 04:37 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: [email protected] <[email protected]>; vignesh C <[email protected]>; Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Dec 1, 2021, at 8:48 PM, Amit Kapila <[email protected]> wrote:
>
> The patch disables the subscription for non-transient errors. I am not
> sure if we can easily make the call to decide whether any particular
> error is transient or not. For example, DISK_FULL or OUT_OF_MEMORY
> might not rectify itself. Why not just allow to disable the
> subscription on any error? And then let the user check the error
> either in view or logs and decide whether it would like to enable the
> subscription or do something before it (like making space in disk, or
> fixing the network).
The original idea of the patch, back when I first wrote and proposed it, was to remove the *absurdity* of retrying a transaction which, in the absence of human intervention, was guaranteed to simply fail again ad infinitum. Retrying in the face of resource errors is not *absurd* even though it might fail again ad infinitum. The reason is that there is at least a chance that the situation will clear up without human intervention.
> The other problem I see with this transient error stuff is maintaining
> the list of error codes that we think are transient. I think we need a
> discussion for each of the error_codes we are listing now and whatever
> new error_code we add in the future which doesn't seem like a good
> idea.
A reasonable rule might be: "the subscription will be disabled if the server can determine that retries cannot possibly succeed without human intervention." We shouldn't need to categorize all error codes perfectly, as long as we're conservative. What I propose is similar to how we determine whether to mark a function leakproof; we don't have to mark all leakproof functions as such, we just can't mark one as such if it is not.
If we're going to debate the error codes, I think we would start with an empty list, and add to the list on sufficient analysis.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-06 06:56 [email protected] <[email protected]>
parent: Mark Dilger <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: [email protected] @ 2021-12-06 06:56 UTC (permalink / raw)
To: 'Mark Dilger' <[email protected]>; Amit Kapila <[email protected]>; +Cc: vignesh C <[email protected]>; Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Monday, December 6, 2021 1:38 PM Mark Dilger <[email protected]> wrote:
> > On Dec 1, 2021, at 8:48 PM, Amit Kapila <[email protected]> wrote:
> >
> > The patch disables the subscription for non-transient errors. I am not
> > sure if we can easily make the call to decide whether any particular
> > error is transient or not. For example, DISK_FULL or OUT_OF_MEMORY
> > might not rectify itself. Why not just allow to disable the
> > subscription on any error? And then let the user check the error
> > either in view or logs and decide whether it would like to enable the
> > subscription or do something before it (like making space in disk, or
> > fixing the network).
>
> The original idea of the patch, back when I first wrote and proposed it, was to
> remove the *absurdity* of retrying a transaction which, in the absence of
> human intervention, was guaranteed to simply fail again ad infinitum.
> Retrying in the face of resource errors is not *absurd* even though it might fail
> again ad infinitum. The reason is that there is at least a chance that the
> situation will clear up without human intervention.
In my humble opinion, I felt the original purpose of the patch was to partially remedy
the situation that during the failure of apply, the apply process keeps going
into the infinite error loop.
I'd say that in this sense, if we include such resource errors, we fail to achieve
the purpose in some cases, because of some left possibilities of infinite loop.
Disabling the subscription with even one any error excludes this irregular possibility,
since there's no room to continue the infinite loop.
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-06 10:52 [email protected] <[email protected]>
parent: Greg Nancarrow <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: [email protected] @ 2021-12-06 10:52 UTC (permalink / raw)
To: 'Greg Nancarrow' <[email protected]>; +Cc: Amit Kapila <[email protected]>; vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Monday, December 6, 2021 1:16 PM Greg Nancarrow <[email protected]> wrote:
> On Sat, Dec 4, 2021 at 12:20 AM [email protected]
> <[email protected]> wrote:
> >
> > Hi, I've made a new patch v11 that incorporated suggestions described
> above.
> >
>
> Some review comments for the v11 patch:
Thank you for your reviews !
> doc/src/sgml/ref/create_subscription.sgml
> (1) Possible wording improvement?
>
> BEFORE:
> + Specifies whether the subscription should be automatically disabled
> + if replicating data from the publisher triggers errors. The default
> + is <literal>false</literal>.
> AFTER:
> + Specifies whether the subscription should be automatically disabled
> + if any errors are detected by subscription workers during data
> + replication from the publisher. The default is <literal>false</literal>.
Fixed.
> src/backend/replication/logical/worker.c
> (2) WorkerErrorRecovery comments
> Instead of:
>
> + * As a preparation for disabling the subscription, emit the error,
> + * handle the transaction and clean up the memory context of
> + * error. ErrorContext is reset by FlushErrorState.
>
> why not just say:
>
> + Worker error recovery processing, in preparation for disabling the
> + subscription.
>
> And then comment the function's code lines:
>
> e.g.
>
> /* Emit the error */
> ...
> /* Abort any active transaction */
> ...
> /* Reset the ErrorContext */
> ...
Agreed. Fixed.
> (3) DisableSubscriptionOnError return
>
> The "if (!subform->subdisableonerr)" block should probably first:
> heap_freetuple(tup);
>
> (regardless of the fact the only current caller will proc_exit anyway)
Fixed.
> (4) did_error flag
>
> I think perhaps the previously-used flag name "disable_subscription"
> is better, or maybe "error_recovery_done".
> Also, I think it would look better if it was set AFTER
> WorkerErrorRecovery() was called.
Adopted error_recovery_done
and changed its places accordingly.
> (5) DisableSubscriptionOnError LOG message
>
> This version of the patch removes the LOG message:
>
> + ereport(LOG,
> + errmsg("logical replication subscription \"%s\" will be disabled due
> to error: %s",
> + MySubscription->name, edata->message));
>
> Perhaps a similar error message could be logged prior to EmitErrorReport()?
>
> e.g.
> "logical replication subscription \"%s\" will be disabled due to an error"
Added.
I've attached the new version v12.
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v12-0001-Optionally-disable-subscriptions-on-error.patch (49.9K, ../../TYCPR01MB83737F77BBE63432611CE7C8ED6D9@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v12-0001-Optionally-disable-subscriptions-on-error.patch)
download | inline diff:
From 62df69d31191f4b971f4a6f824f9e011c50e94fc Mon Sep 17 00:00:00 2001
From: Takamichi Osumi <[email protected]>
Date: Mon, 6 Dec 2021 10:18:29 +0000
Subject: [PATCH v12] Optionally disable subscriptions on error
Logical replication apply workers for a subscription can easily get
stuck in an infinite loop of attempting to apply a change,
triggering an error (such as a constraint violation), exiting with
an error written to the subscription worker log, and restarting.
To partially remedy the situation, adding a new
subscription_parameter named 'disable_on_error'. To be consistent
with old behavior, the parameter defaults to false. When true, both
the table sync worker and apply worker catch any errors thrown and
disable the subscription in order to break the loop. The error is
still also written to the logs.
Require to bump catalog version.
Proposed and written originally by Mark Dilger
Taken over by Osumi Takamichi, Greg Nancarrow
Reviewed by Greg Nancarrow, Vignesh C, Amit Kapila
Discussion : https://www.postgresql.org/message-id/DB35438F-9356-4841-89A0-412709EBD3AB%40enterprisedb.com
---
doc/src/sgml/catalogs.sgml | 10 ++
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 11 ++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 27 +++-
src/backend/replication/logical/launcher.c | 1 +
src/backend/replication/logical/worker.c | 155 ++++++++++++++++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 10 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 4 +
src/test/regress/expected/subscription.out | 119 ++++++++------
src/test/regress/sql/subscription.sql | 14 ++
src/test/subscription/t/027_disable_on_error.pl | 203 ++++++++++++++++++++++++
16 files changed, 513 insertions(+), 70 deletions(-)
create mode 100644 src/test/subscription/t/027_disable_on_error.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index c1d11be..e879796 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7713,6 +7713,16 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<row>
<entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdisableonerr</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription will be disabled when subscription
+ worker detects any errors
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
</para>
<para>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 0b027cc..3109ee9 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -201,8 +201,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
information. The parameters that can be altered
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
- <literal>binary</literal>, and
- <literal>streaming</literal>.
+ <literal>binary</literal>,<literal>streaming</literal>, and
+ <literal>disable_on_error</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 990a41f..186f022 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -142,6 +142,17 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</varlistentry>
<varlistentry>
+ <term><literal>disable_on_error</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription should be automatically disabled
+ if any errors are detected by subscription workers during data
+ replication from the publisher. The default is <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>enabled</literal> (<type>boolean</type>)</term>
<listitem>
<para>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 25021e2..9b416dd 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -69,6 +69,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->binary = subform->subbinary;
sub->stream = subform->substream;
sub->twophasestate = subform->subtwophasestate;
+ sub->disableonerr = subform->subdisableonerr;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 61b515c..41f61bf 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1259,7 +1259,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
- substream, subtwophasestate, subslotname, subsynccommit, subpublications)
+ substream, subtwophasestate, subdisableonerr, subslotname, subsynccommit, subpublications)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_workers AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9427e86..5cb6ca7 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -61,6 +61,7 @@
#define SUBOPT_BINARY 0x00000080
#define SUBOPT_STREAMING 0x00000100
#define SUBOPT_TWOPHASE_COMMIT 0x00000200
+#define SUBOPT_DISABLE_ON_ERR 0x00000400
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -82,6 +83,7 @@ typedef struct SubOpts
bool binary;
bool streaming;
bool twophase;
+ bool disableonerr;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -129,6 +131,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->streaming = false;
if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT))
opts->twophase = false;
+ if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
+ opts->disableonerr = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -248,6 +252,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_TWOPHASE_COMMIT;
opts->twophase = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR) &&
+ strcmp(defel->defname, "disable_on_error") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DISABLE_ON_ERR))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DISABLE_ON_ERR;
+ opts->disableonerr = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -397,7 +410,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
supported_opts = (SUBOPT_CONNECT | SUBOPT_ENABLED | SUBOPT_CREATE_SLOT |
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT);
+ SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
+ SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -471,6 +485,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
CharGetDatum(opts.twophase ?
LOGICALREP_TWOPHASE_STATE_PENDING :
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -871,7 +886,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
{
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING);
+ SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -920,6 +935,14 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_substream - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR))
+ {
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = BoolGetDatum(opts.disableonerr);
+ replaces[Anum_pg_subscription_subdisableonerr - 1]
+ = true;
+ }
+
update_tuple = true;
break;
}
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 3fb4caa..febfc4d 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -132,6 +132,7 @@ get_subscription_list(void)
sub->dbid = subform->subdbid;
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->name = pstrdup(NameStr(subform->subname));
/* We don't fill fields we are not interested in. */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 2e79302..0aef7e5 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -136,6 +136,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/partition.h"
#include "catalog/pg_inherits.h"
@@ -2755,6 +2756,92 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/*
+ * Worker error recovery processing, in preparation for disabling the
+ * subscription.
+ */
+static void
+WorkerErrorRecovery(void)
+{
+ /* Notify the subscription will be no longer valid */
+ ereport(LOG,
+ errmsg("logical replication subscription \"%s\" will be disabled due to an error",
+ MySubscription->name));
+ /* Emit the error */
+ EmitErrorReport();
+ /* Abort any active transaction */
+ AbortOutOfAnyTransaction();
+ /* Reset the ErrorContext */
+ FlushErrorState();
+}
+
+/*
+ * Recover from a possibly aborted transaction state and disable the current
+ * subscription.
+ */
+static void
+DisableSubscriptionOnError(void)
+{
+ Relation rel;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Form_pg_subscription subform;
+
+ /* Disable the subscription in a fresh transaction */
+ StartTransactionCommand();
+
+ /* Look up our subscription in the catalogs */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy2(SUBSCRIPTIONNAME, MyDatabaseId,
+ CStringGetDatum(MySubscription->name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("subscription \"%s\" does not exist",
+ MySubscription->name));
+
+ subform = (Form_pg_subscription) GETSTRUCT(tup);
+ LockSharedObject(SubscriptionRelationId, subform->oid, 0, AccessExclusiveLock);
+
+ /*
+ * We would not be here unless this subscription's disableonerr field was
+ * true when our worker began applying changes, but check whether that
+ * field has changed in the interim.
+ */
+ if (!subform->subdisableonerr)
+ {
+ /*
+ * Disabling the subscription has been done already. No need of
+ * additional work.
+ */
+ heap_freetuple(tup);
+ table_close(rel, RowExclusiveLock);
+ CommitTransactionCommand();
+ return;
+ }
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Set the subscription to disabled, and note the reason. */
+ values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(false);
+ replaces[Anum_pg_subscription_subenabled - 1] = true;
+
+ /* Update the catalog */
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+ heap_freetuple(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ CommitTransactionCommand();
+}
+
+/*
* Send a Standby Status Update message to server.
*
* 'recvpos' is the latest LSN we've received data to, force is set if we need
@@ -3339,6 +3426,7 @@ ApplyWorkerMain(Datum main_arg)
char *myslotname;
WalRcvStreamOptions options;
int server_version;
+ bool error_recovery_done = false;
/* Attach to slot */
logicalrep_worker_attach(worker_slot);
@@ -3453,15 +3541,33 @@ ApplyWorkerMain(Datum main_arg)
0, /* message type */
InvalidTransactionId,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- PG_RE_THROW();
+
+ if (MySubscription->disableonerr)
+ {
+ WorkerErrorRecovery();
+ error_recovery_done = true;
+ }
+ else
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
- /* allocate slot name in long-lived context */
- myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
+ /* If we caught an error above, disable the subscription */
+ if (error_recovery_done)
+ {
+ DisableSubscriptionOnError();
+ proc_exit(0);
+ }
+ else
+ {
+ /* allocate slot name in long-lived context */
+ myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
- pfree(syncslotname);
+ pfree(syncslotname);
+ }
}
else
{
@@ -3580,10 +3686,11 @@ ApplyWorkerMain(Datum main_arg)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(cctx);
+
/* report the apply error */
if (apply_error_callback_arg.command != 0)
{
- MemoryContext ecxt = MemoryContextSwitchTo(cctx);
ErrorData *errdata = CopyErrorData();
pgstat_report_subworker_error(MyLogicalRepWorker->subid,
@@ -3594,13 +3701,43 @@ ApplyWorkerMain(Datum main_arg)
apply_error_callback_arg.command,
apply_error_callback_arg.remote_xid,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- }
- PG_RE_THROW();
+ if (MySubscription->disableonerr)
+ {
+ WorkerErrorRecovery();
+ error_recovery_done = true;
+ }
+ else
+ {
+ /*
+ * Some work in error recovery work is done. Switch to the old
+ * memory context and rethrow.
+ */
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ }
+ else
+ {
+ /*
+ * Don't miss any error, even when it's not reported to stats
+ * collector.
+ */
+ if (MySubscription->disableonerr)
+ {
+ WorkerErrorRecovery();
+ error_recovery_done = true;
+ }
+ else
+ /* Simply rethrow because of no recovery work */
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
+ if (error_recovery_done)
+ DisableSubscriptionOnError();
+
proc_exit(0);
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index c590003..610f85a 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4452,6 +4452,7 @@ getSubscriptions(Archive *fout)
int i_rolname;
int i_substream;
int i_subtwophasestate;
+ int i_subdisableonerr;
int i_subconninfo;
int i_subslotname;
int i_subsynccommit;
@@ -4500,12 +4501,18 @@ getSubscriptions(Archive *fout)
appendPQExpBufferStr(query, " false AS substream,\n");
if (fout->remoteVersion >= 150000)
- appendPQExpBufferStr(query, " s.subtwophasestate\n");
+ appendPQExpBufferStr(query, " s.subtwophasestate,\n");
else
appendPQExpBuffer(query,
- " '%c' AS subtwophasestate\n",
+ " '%c' AS subtwophasestate,\n",
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBuffer(query, " s.subdisableonerr\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdisableonerr\n");
+
appendPQExpBufferStr(query,
"FROM pg_subscription s\n"
"WHERE s.subdbid = (SELECT oid FROM pg_database\n"
@@ -4526,6 +4533,7 @@ getSubscriptions(Archive *fout)
i_subbinary = PQfnumber(res, "subbinary");
i_substream = PQfnumber(res, "substream");
i_subtwophasestate = PQfnumber(res, "subtwophasestate");
+ i_subdisableonerr = PQfnumber(res, "subdisableonerr");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4553,6 +4561,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_substream));
subinfo[i].subtwophasestate =
pg_strdup(PQgetvalue(res, i, i_subtwophasestate));
+ subinfo[i].subdisableonerr =
+ pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
if (strlen(subinfo[i].rolname) == 0)
pg_log_warning("owner of subscription \"%s\" appears to be invalid",
@@ -4625,6 +4635,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0)
appendPQExpBufferStr(query, ", two_phase = on");
+ if (strcmp(subinfo->subdisableonerr, "f") != 0)
+ appendPQExpBufferStr(query, ", disable_on_error = on");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index d1d8608..0512111 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -655,6 +655,7 @@ typedef struct _SubscriptionInfo
char *subbinary;
char *substream;
char *subtwophasestate;
+ char *subdisableonerr;
char *subsynccommit;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index ea721d9..e2521a2 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6508,7 +6508,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false};
+ false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6542,11 +6542,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Binary"),
gettext_noop("Streaming"));
- /* Two_phase is only supported in v15 and higher */
+ /* Two_phase and disable_on_error is only supported in v15 and higher */
if (pset.sversion >= 150000)
appendPQExpBuffer(&buf,
- ", subtwophasestate AS \"%s\"\n",
- gettext_noop("Two phase commit"));
+ ", subtwophasestate AS \"%s\"\n"
+ ", subdisableonerr AS \"%s\"\n",
+ gettext_noop("Two phase commit"),
+ gettext_noop("Disable On Error"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 2f412ca..5497290 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1681,7 +1681,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit", "disable_on_error");
/* ALTER SUBSCRIPTION <name> SET PUBLICATION */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "PUBLICATION"))
{
@@ -2939,7 +2939,7 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
"enabled", "slot_name", "streaming",
- "synchronous_commit", "two_phase");
+ "synchronous_commit", "two_phase", "disable_on_error");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 2106149..284a90d 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -67,6 +67,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
char subtwophasestate; /* Stream two-phase transactions */
+ bool subdisableonerr; /* True if apply errors should disable the
+ * subscription upon error */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -103,6 +106,7 @@ typedef struct Subscription
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
char twophasestate; /* Allow streaming two-phase transactions */
+ bool disableonerr; /* Whether errors automatically disable */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 15a1ac6..db0bd19 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -94,10 +94,10 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
@@ -129,10 +129,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -165,19 +165,19 @@ ERROR: binary requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -188,19 +188,19 @@ ERROR: streaming requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication already exists
@@ -215,10 +215,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication used more then once
@@ -233,10 +233,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -270,10 +270,10 @@ ERROR: two_phase requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | f | off | dbname=regress_doesnotexist
(1 row)
--fail - alter of two_phase option not supported.
@@ -282,10 +282,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -294,10 +294,33 @@ DROP SUBSCRIPTION regress_testsub;
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+ERROR: disable_on_error requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | t | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7faa935..ee4a39b 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -228,6 +228,20 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+
+\dRs+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/t/027_disable_on_error.pl b/src/test/subscription/t/027_disable_on_error.pl
new file mode 100644
index 0000000..1104a50
--- /dev/null
+++ b/src/test/subscription/t/027_disable_on_error.pl
@@ -0,0 +1,203 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+
+# Wait for the named subscriptions to catch up or to be disabled.
+sub wait_for_subscriptions
+{
+ my ($node_name, $dbname, @subscriptions) = @_;
+
+ # Unique-ify the subscriptions passed by the caller
+ my %unique = map { $_ => 1 } @subscriptions;
+ my @unique = sort keys %unique;
+ my $unique_count = scalar(@unique);
+
+ # Construct a SQL list from the unique subscription names
+ my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
+ my $sublist = join(', ', map { "'$_'" } @escaped);
+
+ my $polling_sql = qq(
+ SELECT COUNT(1) = $unique_count FROM
+ (SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ LEFT JOIN pg_catalog.pg_subscription_rel sr
+ ON sr.srsubid = s.oid
+ WHERE (sr IS NULL OR sr.srsubstate IN ('s', 'r'))
+ AND s.subname IN ($sublist)
+ AND s.subenabled IS TRUE
+ UNION
+ SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ WHERE s.subname IN ($sublist)
+ AND s.subenabled IS FALSE
+ ) AS synced_or_disabled
+ );
+ return $node_name->poll_query_until($dbname, $polling_sql);
+}
+
+my @schemas = qw(s1 s2);
+my ($schema, $cmd);
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+# Create identical schema, table and index on both the publisher and
+# subscriber
+for $schema (@schemas)
+{
+ $cmd = qq(
+CREATE SCHEMA $schema;
+CREATE TABLE $schema.tbl (i INT);
+ALTER TABLE $schema.tbl REPLICA IDENTITY FULL;
+CREATE INDEX ${schema}_tbl_idx ON $schema.tbl(i));
+ $node_publisher->safe_psql('postgres', $cmd);
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Create non-unique data in both schemas on the publisher.
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (1), (1), (1));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Create an additional unique index in schema s1 on the subscriber only. When
+# we create subscriptions, below, this should cause subscription "s1" on the
+# subscriber to fail during initial synchronization and to get automatically
+# disabled.
+$cmd = qq(CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Create publications and subscriptions linking the schemas on
+# the publisher with those on the subscriber. This tests that the
+# uniqueness violations cause subscription "s1" to fail during
+# initial synchronization.
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+for $schema (@schemas)
+{
+ # Create the publication for this table
+ $cmd = qq(
+CREATE PUBLICATION $schema FOR TABLE $schema.tbl);
+ $node_publisher->safe_psql('postgres', $cmd);
+
+ # Create the subscription for this table
+ $cmd = qq(
+CREATE SUBSCRIPTION $schema
+ CONNECTION '$publisher_connstr'
+ PUBLICATION $schema
+ WITH (disable_on_error = true));
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Wait for the initial subscription synchronizations to finish or fail.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT i, COUNT(*) FROM s2.tbl GROUP BY i);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "1|3", "subscription s2 replicated initial data");
+
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (2));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that the
+# problems encountered by subscription "s1" do not cause subscription "s2" to
+# get stuck.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should still be disabled and have replicated no data
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 still disabled");
+
+# Subscription "s2" should still be enabled and have replicated all changes
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s2 replicated data");
+
+# Drop the unique index on "s1" which caused the subscription to be disabled
+$cmd = qq(DROP INDEX s1.s1_tbl_unique);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Re-enable the subscription "s1"
+$cmd = q(ALTER SUBSCRIPTION s1 ENABLE);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Wait for the data to replicate
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check that we have the new data in s1.tbl
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s1.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s1 replicated data");
+
+# Delete the data from the subscriber only, and recreate the unique index
+$cmd = q(
+DELETE FROM s1.tbl;
+CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Add more non-unique data to the publisher
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (3), (3), (3));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that
+# uniqueness violations encountered during replication cause s1 to be disabled.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "3|7", "subscription s2 replicated additional data");
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-06 10:59 Amit Kapila <[email protected]>
parent: Mark Dilger <[email protected]>
1 sibling, 0 replies; 74+ messages in thread
From: Amit Kapila @ 2021-12-06 10:59 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: [email protected] <[email protected]>; vignesh C <[email protected]>; Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Dec 6, 2021 at 10:07 AM Mark Dilger
<[email protected]> wrote:
>
> > On Dec 1, 2021, at 8:48 PM, Amit Kapila <[email protected]> wrote:
> >
> > The patch disables the subscription for non-transient errors. I am not
> > sure if we can easily make the call to decide whether any particular
> > error is transient or not. For example, DISK_FULL or OUT_OF_MEMORY
> > might not rectify itself. Why not just allow to disable the
> > subscription on any error? And then let the user check the error
> > either in view or logs and decide whether it would like to enable the
> > subscription or do something before it (like making space in disk, or
> > fixing the network).
>
> The original idea of the patch, back when I first wrote and proposed it, was to remove the *absurdity* of retrying a transaction which, in the absence of human intervention, was guaranteed to simply fail again ad infinitum. Retrying in the face of resource errors is not *absurd* even though it might fail again ad infinitum. The reason is that there is at least a chance that the situation will clear up without human intervention.
>
> > The other problem I see with this transient error stuff is maintaining
> > the list of error codes that we think are transient. I think we need a
> > discussion for each of the error_codes we are listing now and whatever
> > new error_code we add in the future which doesn't seem like a good
> > idea.
>
> A reasonable rule might be: "the subscription will be disabled if the server can determine that retries cannot possibly succeed without human intervention." We shouldn't need to categorize all error codes perfectly, as long as we're conservative. What I propose is similar to how we determine whether to mark a function leakproof; we don't have to mark all leakproof functions as such, we just can't mark one as such if it is not.
>
> If we're going to debate the error codes, I think we would start with an empty list, and add to the list on sufficient analysis.
>
Yeah, an empty list is a sort of what I thought was a good start
point. I feel we should learn from real-world use cases to see if
people really want to continue retrying even after using this option.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-06 16:06 Mark Dilger <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Mark Dilger @ 2021-12-06 16:06 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Amit Kapila <[email protected]>; vignesh C <[email protected]>; Greg Nancarrow <[email protected]>; Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Dec 5, 2021, at 10:56 PM, [email protected] wrote:
>
> In my humble opinion, I felt the original purpose of the patch was to partially remedy
> the situation that during the failure of apply, the apply process keeps going
> into the infinite error loop.
I agree.
> I'd say that in this sense, if we include such resource errors, we fail to achieve
> the purpose in some cases, because of some left possibilities of infinite loop.
> Disabling the subscription with even one any error excludes this irregular possibility,
> since there's no room to continue the infinite loop.
I don't think there is any right answer here. It's a question of policy preferences.
My concern about disabling a subscription in response to *any* error is that people may find the feature does more harm than good. Disabling the subscription in response to an occasional deadlock against other database users, or occasional resource pressure, might annoy people and lead to the feature simply not being used.
I am happy to defer to your policy preference. Thanks for your work on the patch!
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-07 00:22 Greg Nancarrow <[email protected]>
parent: Mark Dilger <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Greg Nancarrow @ 2021-12-07 00:22 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: [email protected] <[email protected]>; Amit Kapila <[email protected]>; vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Dec 7, 2021 at 3:06 AM Mark Dilger <[email protected]> wrote:
>
> My concern about disabling a subscription in response to *any* error is that people may find the feature does more harm than good. Disabling the subscription in response to an occasional deadlock against other database users, or occasional resource pressure, might annoy people and lead to the feature simply not being used.
>
I can understand this point of view.
It kind of suggests to me the possibility of something like a
configurable timeout (e.g. disable the subscription if the same error
has occurred for more than X minutes) or, similarly, perhaps if some
threshold has been reached (e.g. same error has occurred more than X
times), but I think that this was previously suggested by Peter Smith
and the idea wasn't looked upon all that favorably?
Regards,
Greg Nancarrow
Fujitsu Australia
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-08 13:10 Amit Kapila <[email protected]>
parent: Greg Nancarrow <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Amit Kapila @ 2021-12-08 13:10 UTC (permalink / raw)
To: Greg Nancarrow <[email protected]>; +Cc: Mark Dilger <[email protected]>; [email protected] <[email protected]>; vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Dec 7, 2021 at 5:52 AM Greg Nancarrow <[email protected]> wrote:
>
> On Tue, Dec 7, 2021 at 3:06 AM Mark Dilger <[email protected]> wrote:
> >
> > My concern about disabling a subscription in response to *any* error is that people may find the feature does more harm than good. Disabling the subscription in response to an occasional deadlock against other database users, or occasional resource pressure, might annoy people and lead to the feature simply not being used.
> >
> I can understand this point of view.
> It kind of suggests to me the possibility of something like a
> configurable timeout (e.g. disable the subscription if the same error
> has occurred for more than X minutes) or, similarly, perhaps if some
> threshold has been reached (e.g. same error has occurred more than X
> times), but I think that this was previously suggested by Peter Smith
> and the idea wasn't looked upon all that favorably?
>
I think if we are really worried about transient errors then probably
the idea "disable only if the same error has occurred more than X
times" seems preferable as compared to taking a decision on which
error_codes fall in the transient error category.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-08 15:52 Mark Dilger <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Mark Dilger @ 2021-12-08 15:52 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Greg Nancarrow <[email protected]>; [email protected] <[email protected]>; vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Dec 8, 2021, at 5:10 AM, Amit Kapila <[email protected]> wrote:
>
> I think if we are really worried about transient errors then probably
> the idea "disable only if the same error has occurred more than X
> times" seems preferable as compared to taking a decision on which
> error_codes fall in the transient error category.
No need. We can revisit this design decision in a later release cycle if the current patch's design proves problematic in the field.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-09 04:09 Amit Kapila <[email protected]>
parent: Mark Dilger <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Amit Kapila @ 2021-12-09 04:09 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Greg Nancarrow <[email protected]>; [email protected] <[email protected]>; vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Dec 8, 2021 at 9:22 PM Mark Dilger <[email protected]> wrote:
>
>
> > On Dec 8, 2021, at 5:10 AM, Amit Kapila <[email protected]> wrote:
> >
> > I think if we are really worried about transient errors then probably
> > the idea "disable only if the same error has occurred more than X
> > times" seems preferable as compared to taking a decision on which
> > error_codes fall in the transient error category.
>
> No need. We can revisit this design decision in a later release cycle if the current patch's design proves problematic in the field.
>
So, do you agree that we can disable the subscription on any error if
this parameter is set?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-09 16:44 Mark Dilger <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 0 replies; 74+ messages in thread
From: Mark Dilger @ 2021-12-09 16:44 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Greg Nancarrow <[email protected]>; [email protected] <[email protected]>; vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Dec 8, 2021, at 8:09 PM, Amit Kapila <[email protected]> wrote:
>
> So, do you agree that we can disable the subscription on any error if
> this parameter is set?
Yes, I think that is fine. We can commit it that way, and revisit the issue for v16 if it becomes a problem in practice.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-13 09:57 vignesh C <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: vignesh C @ 2021-12-13 09:57 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Greg Nancarrow <[email protected]>; Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Dec 6, 2021 at 4:22 PM [email protected]
<[email protected]> wrote:
>
> On Monday, December 6, 2021 1:16 PM Greg Nancarrow <[email protected]> wrote:
> > On Sat, Dec 4, 2021 at 12:20 AM [email protected]
> > <[email protected]> wrote:
> > >
> > > Hi, I've made a new patch v11 that incorporated suggestions described
> > above.
> > >
> >
> > Some review comments for the v11 patch:
> Thank you for your reviews !
>
> > doc/src/sgml/ref/create_subscription.sgml
> > (1) Possible wording improvement?
> >
> > BEFORE:
> > + Specifies whether the subscription should be automatically disabled
> > + if replicating data from the publisher triggers errors. The default
> > + is <literal>false</literal>.
> > AFTER:
> > + Specifies whether the subscription should be automatically disabled
> > + if any errors are detected by subscription workers during data
> > + replication from the publisher. The default is <literal>false</literal>.
> Fixed.
>
> > src/backend/replication/logical/worker.c
> > (2) WorkerErrorRecovery comments
> > Instead of:
> >
> > + * As a preparation for disabling the subscription, emit the error,
> > + * handle the transaction and clean up the memory context of
> > + * error. ErrorContext is reset by FlushErrorState.
> >
> > why not just say:
> >
> > + Worker error recovery processing, in preparation for disabling the
> > + subscription.
> >
> > And then comment the function's code lines:
> >
> > e.g.
> >
> > /* Emit the error */
> > ...
> > /* Abort any active transaction */
> > ...
> > /* Reset the ErrorContext */
> > ...
> Agreed. Fixed.
>
> > (3) DisableSubscriptionOnError return
> >
> > The "if (!subform->subdisableonerr)" block should probably first:
> > heap_freetuple(tup);
> >
> > (regardless of the fact the only current caller will proc_exit anyway)
> Fixed.
>
> > (4) did_error flag
> >
> > I think perhaps the previously-used flag name "disable_subscription"
> > is better, or maybe "error_recovery_done".
> > Also, I think it would look better if it was set AFTER
> > WorkerErrorRecovery() was called.
> Adopted error_recovery_done
> and changed its places accordingly.
>
> > (5) DisableSubscriptionOnError LOG message
> >
> > This version of the patch removes the LOG message:
> >
> > + ereport(LOG,
> > + errmsg("logical replication subscription \"%s\" will be disabled due
> > to error: %s",
> > + MySubscription->name, edata->message));
> >
> > Perhaps a similar error message could be logged prior to EmitErrorReport()?
> >
> > e.g.
> > "logical replication subscription \"%s\" will be disabled due to an error"
> Added.
>
> I've attached the new version v12.
Thanks for the updated patch, few comments:
1) This is not required as it is not used in the caller.
+++ b/src/backend/replication/logical/launcher.c
@@ -132,6 +132,7 @@ get_subscription_list(void)
sub->dbid = subform->subdbid;
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->disableonerr = subform->subdisableonerr;
sub->name = pstrdup(NameStr(subform->subname));
/* We don't fill fields we are not interested in. */
2) Should this be changed:
+ <structfield>subdisableonerr</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription will be disabled when subscription
+ worker detects any errors
+ </para></entry>
+ </row>
To:
+ <structfield>subdisableonerr</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription will be disabled when subscription's
+ worker detects any errors
+ </para></entry>
+ </row>
3) The last line can be slightly adjusted to keep within 80 chars:
+ Specifies whether the subscription should be automatically disabled
+ if any errors are detected by subscription workers during data
+ replication from the publisher. The default is
<literal>false</literal>.
4) Similarly this too can be handled:
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1259,7 +1259,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
- substream, subtwophasestate, subslotname,
subsynccommit, subpublications)
+ substream, subtwophasestate, subdisableonerr,
subslotname, subsynccommit, subpublications)
ON pg_subscription TO public;
5) Since disabling subscription code is common in and else, can we
move it below:
+ if (MySubscription->disableonerr)
+ {
+ WorkerErrorRecovery();
+ error_recovery_done = true;
+ }
+ else
+ {
+ /*
+ * Some work in error recovery work is
done. Switch to the old
+ * memory context and rethrow.
+ */
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ }
+ else
+ {
+ /*
+ * Don't miss any error, even when it's not
reported to stats
+ * collector.
+ */
+ if (MySubscription->disableonerr)
+ {
+ WorkerErrorRecovery();
+ error_recovery_done = true;
+ }
+ else
+ /* Simply rethrow because of no recovery work */
+ PG_RE_THROW();
+ }
6) Can we move LockSharedObject below the if condition.
+ subform = (Form_pg_subscription) GETSTRUCT(tup);
+ LockSharedObject(SubscriptionRelationId, subform->oid, 0,
AccessExclusiveLock);
+
+ /*
+ * We would not be here unless this subscription's
disableonerr field was
+ * true when our worker began applying changes, but check whether that
+ * field has changed in the interim.
+ */
+ if (!subform->subdisableonerr)
+ {
+ /*
+ * Disabling the subscription has been done already. No need of
+ * additional work.
+ */
+ heap_freetuple(tup);
+ table_close(rel, RowExclusiveLock);
+ CommitTransactionCommand();
+ return;
+ }
+
Regards,
Vignesh
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-14 05:34 [email protected] <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: [email protected] @ 2021-12-14 05:34 UTC (permalink / raw)
To: 'vignesh C' <[email protected]>; +Cc: Greg Nancarrow <[email protected]>; Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Monday, December 13, 2021 6:57 PM vignesh C <[email protected]> wrote:
> On Mon, Dec 6, 2021 at 4:22 PM [email protected]
> <[email protected]> wrote:
> >
> > I've attached the new version v12.
I appreciate your review.
> Thanks for the updated patch, few comments:
> 1) This is not required as it is not used in the caller.
> +++ b/src/backend/replication/logical/launcher.c
> @@ -132,6 +132,7 @@ get_subscription_list(void)
> sub->dbid = subform->subdbid;
> sub->owner = subform->subowner;
> sub->enabled = subform->subenabled;
> + sub->disableonerr = subform->subdisableonerr;
> sub->name = pstrdup(NameStr(subform->subname));
> /* We don't fill fields we are not interested in. */
Okay.
The comment of the get_subscription_list() mentions that
we collect and fill only fields related to worker start/stop.
Then, I didn't need it. Fixed.
> 2) Should this be changed:
> + <structfield>subdisableonerr</structfield> <type>bool</type>
> + </para>
> + <para>
> + If true, the subscription will be disabled when subscription
> + worker detects any errors
> + </para></entry>
> + </row>
> To:
> + <structfield>subdisableonerr</structfield> <type>bool</type>
> + </para>
> + <para>
> + If true, the subscription will be disabled when subscription's
> + worker detects any errors
> + </para></entry>
> + </row>
I felt either is fine. So fixed.
> 3) The last line can be slightly adjusted to keep within 80 chars:
> + Specifies whether the subscription should be automatically disabled
> + if any errors are detected by subscription workers during data
> + replication from the publisher. The default is
> <literal>false</literal>.
Fixed.
> 4) Similarly this too can be handled:
> --- a/src/backend/catalog/system_views.sql
> +++ b/src/backend/catalog/system_views.sql
> @@ -1259,7 +1259,7 @@ REVOKE ALL ON pg_replication_origin_status FROM
> public;
> -- All columns of pg_subscription except subconninfo are publicly readable.
> REVOKE ALL ON pg_subscription FROM public; GRANT SELECT (oid,
> subdbid, subname, subowner, subenabled, subbinary,
> - substream, subtwophasestate, subslotname,
> subsynccommit, subpublications)
> + substream, subtwophasestate, subdisableonerr,
> subslotname, subsynccommit, subpublications)
> ON pg_subscription TO public;
I split the line into two to make each line less than 80 chars.
> 5) Since disabling subscription code is common in and else, can we move it
> below:
> + if (MySubscription->disableonerr)
> + {
> + WorkerErrorRecovery();
> + error_recovery_done = true;
> + }
> + else
> + {
> + /*
> + * Some work in error recovery work is
> done. Switch to the old
> + * memory context and rethrow.
> + */
> + MemoryContextSwitchTo(ecxt);
> + PG_RE_THROW();
> + }
> + }
> + else
> + {
> + /*
> + * Don't miss any error, even when it's not
> reported to stats
> + * collector.
> + */
> + if (MySubscription->disableonerr)
> + {
> + WorkerErrorRecovery();
> + error_recovery_done = true;
> + }
> + else
> + /* Simply rethrow because of no recovery
> work */
> + PG_RE_THROW();
> + }
I moved the common code below those condition branches.
> 6) Can we move LockSharedObject below the if condition.
> + subform = (Form_pg_subscription) GETSTRUCT(tup);
> + LockSharedObject(SubscriptionRelationId, subform->oid, 0,
> AccessExclusiveLock);
> +
> + /*
> + * We would not be here unless this subscription's
> disableonerr field was
> + * true when our worker began applying changes, but check whether
> that
> + * field has changed in the interim.
> + */
> + if (!subform->subdisableonerr)
> + {
> + /*
> + * Disabling the subscription has been done already. No need
> of
> + * additional work.
> + */
> + heap_freetuple(tup);
> + table_close(rel, RowExclusiveLock);
> + CommitTransactionCommand();
> + return;
> + }
> +
Fixed.
Besides all of those changes, I've removed the obsolete
comment of DisableSubscriptionOnError in v12.
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v13-0001-Optionally-disable-subscriptions-on-error.patch (49.0K, ../../TYCPR01MB8373DEBEB6EC28CB82F50D47ED759@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v13-0001-Optionally-disable-subscriptions-on-error.patch)
download | inline diff:
From a08d2672e78c8184e2bf79d32b41859fa0df954c Mon Sep 17 00:00:00 2001
From: Takamichi Osumi <[email protected]>
Date: Tue, 14 Dec 2021 05:09:41 +0000
Subject: [PATCH v13] Optionally disable subscriptions on error
Logical replication apply workers for a subscription can easily get
stuck in an infinite loop of attempting to apply a change,
triggering an error (such as a constraint violation), exiting with
an error written to the subscription worker log, and restarting.
To partially remedy the situation, adding a new
subscription_parameter named 'disable_on_error'. To be consistent
with old behavior, the parameter defaults to false. When true, both
the table sync worker and apply worker catch any errors thrown and
disable the subscription in order to break the loop. The error is
still also written to the logs.
Require to bump catalog version.
Proposed and written originally by Mark Dilger
Taken over by Osumi Takamichi, Greg Nancarrow
Reviewed by Greg Nancarrow, Vignesh C, Amit Kapila
Discussion : https://www.postgresql.org/message-id/DB35438F-9356-4841-89A0-412709EBD3AB%40enterprisedb.com
---
doc/src/sgml/catalogs.sgml | 10 ++
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 ++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 27 +++-
src/backend/replication/logical/worker.c | 139 +++++++++++++++-
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 10 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 4 +
src/test/regress/expected/subscription.out | 119 ++++++++------
src/test/regress/sql/subscription.sql | 14 ++
src/test/subscription/t/027_disable_on_error.pl | 203 ++++++++++++++++++++++++
15 files changed, 499 insertions(+), 69 deletions(-)
create mode 100644 src/test/subscription/t/027_disable_on_error.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 03e2537..06de805 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7725,6 +7725,16 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<row>
<entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdisableonerr</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription will be disabled when subscription's
+ worker detects any errors
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
</para>
<para>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 0b027cc..3109ee9 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -201,8 +201,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
information. The parameters that can be altered
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
- <literal>binary</literal>, and
- <literal>streaming</literal>.
+ <literal>binary</literal>,<literal>streaming</literal>, and
+ <literal>disable_on_error</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 990a41f..9ca0bb9 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -142,6 +142,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</varlistentry>
<varlistentry>
+ <term><literal>disable_on_error</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription should be automatically disabled
+ if any errors are detected by subscription workers during data
+ replication from the publisher. The default is
+ <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>enabled</literal> (<type>boolean</type>)</term>
<listitem>
<para>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 25021e2..9b416dd 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -69,6 +69,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->binary = subform->subbinary;
sub->stream = subform->substream;
sub->twophasestate = subform->subtwophasestate;
+ sub->disableonerr = subform->subdisableonerr;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 61b515c..9d0ed61 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1259,7 +1259,8 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
- substream, subtwophasestate, subslotname, subsynccommit, subpublications)
+ substream, subtwophasestate, subdisableonerr, subslotname,
+ subsynccommit, subpublications)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_workers AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 2b65808..cebc51a 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -61,6 +61,7 @@
#define SUBOPT_BINARY 0x00000080
#define SUBOPT_STREAMING 0x00000100
#define SUBOPT_TWOPHASE_COMMIT 0x00000200
+#define SUBOPT_DISABLE_ON_ERR 0x00000400
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -82,6 +83,7 @@ typedef struct SubOpts
bool binary;
bool streaming;
bool twophase;
+ bool disableonerr;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -130,6 +132,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->streaming = false;
if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT))
opts->twophase = false;
+ if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
+ opts->disableonerr = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -249,6 +253,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_TWOPHASE_COMMIT;
opts->twophase = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR) &&
+ strcmp(defel->defname, "disable_on_error") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DISABLE_ON_ERR))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DISABLE_ON_ERR;
+ opts->disableonerr = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -390,7 +403,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
supported_opts = (SUBOPT_CONNECT | SUBOPT_ENABLED | SUBOPT_CREATE_SLOT |
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT);
+ SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
+ SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -464,6 +478,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
CharGetDatum(opts.twophase ?
LOGICALREP_TWOPHASE_STATE_PENDING :
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -864,7 +879,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
{
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING);
+ SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -913,6 +928,14 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_substream - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR))
+ {
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = BoolGetDatum(opts.disableonerr);
+ replaces[Anum_pg_subscription_subdisableonerr - 1]
+ = true;
+ }
+
update_tuple = true;
break;
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 2e79302..dc81f85 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -136,6 +136,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/partition.h"
#include "catalog/pg_inherits.h"
@@ -2755,6 +2756,92 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/*
+ * Worker error recovery processing, in preparation for disabling the
+ * subscription.
+ */
+static void
+WorkerErrorRecovery(void)
+{
+ /* Notify the subscription will be no longer valid */
+ ereport(LOG,
+ errmsg("logical replication subscription \"%s\" will be disabled due to an error",
+ MySubscription->name));
+ /* Emit the error */
+ EmitErrorReport();
+ /* Abort any active transaction */
+ AbortOutOfAnyTransaction();
+ /* Reset the ErrorContext */
+ FlushErrorState();
+}
+
+/*
+ * Disable the current subscription.
+ */
+static void
+DisableSubscriptionOnError(void)
+{
+ Relation rel;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Form_pg_subscription subform;
+
+ /* Disable the subscription in a fresh transaction */
+ StartTransactionCommand();
+
+ /* Look up our subscription in the catalogs */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy2(SUBSCRIPTIONNAME, MyDatabaseId,
+ CStringGetDatum(MySubscription->name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("subscription \"%s\" does not exist",
+ MySubscription->name));
+
+ subform = (Form_pg_subscription) GETSTRUCT(tup);
+
+ /*
+ * We would not be here unless this subscription's disableonerr field was
+ * true when our worker began applying changes, but check whether that
+ * field has changed in the interim.
+ */
+ if (!subform->subdisableonerr)
+ {
+ /*
+ * Disabling the subscription has been done already. No need of
+ * additional work.
+ */
+ heap_freetuple(tup);
+ table_close(rel, RowExclusiveLock);
+ CommitTransactionCommand();
+ return;
+ }
+
+ LockSharedObject(SubscriptionRelationId, subform->oid, 0, AccessExclusiveLock);
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Set the subscription to disabled, and note the reason. */
+ values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(false);
+ replaces[Anum_pg_subscription_subenabled - 1] = true;
+
+ /* Update the catalog */
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+ heap_freetuple(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ CommitTransactionCommand();
+}
+
+/*
* Send a Standby Status Update message to server.
*
* 'recvpos' is the latest LSN we've received data to, force is set if we need
@@ -3339,6 +3426,7 @@ ApplyWorkerMain(Datum main_arg)
char *myslotname;
WalRcvStreamOptions options;
int server_version;
+ bool error_recovery_done = false;
/* Attach to slot */
logicalrep_worker_attach(worker_slot);
@@ -3453,15 +3541,33 @@ ApplyWorkerMain(Datum main_arg)
0, /* message type */
InvalidTransactionId,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- PG_RE_THROW();
+
+ if (MySubscription->disableonerr)
+ {
+ WorkerErrorRecovery();
+ error_recovery_done = true;
+ }
+ else
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
- /* allocate slot name in long-lived context */
- myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
+ /* If we caught an error above, disable the subscription */
+ if (error_recovery_done)
+ {
+ DisableSubscriptionOnError();
+ proc_exit(0);
+ }
+ else
+ {
+ /* allocate slot name in long-lived context */
+ myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
- pfree(syncslotname);
+ pfree(syncslotname);
+ }
}
else
{
@@ -3580,10 +3686,11 @@ ApplyWorkerMain(Datum main_arg)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(cctx);
+
/* report the apply error */
if (apply_error_callback_arg.command != 0)
{
- MemoryContext ecxt = MemoryContextSwitchTo(cctx);
ErrorData *errdata = CopyErrorData();
pgstat_report_subworker_error(MyLogicalRepWorker->subid,
@@ -3594,13 +3701,29 @@ ApplyWorkerMain(Datum main_arg)
apply_error_callback_arg.command,
apply_error_callback_arg.remote_xid,
errdata->message);
- MemoryContextSwitchTo(ecxt);
+
+ if (!MySubscription->disableonerr)
+ {
+ /*
+ * Some work in error recovery work is done. Switch to the old
+ * memory context and rethrow.
+ */
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
+ else if (!MySubscription->disableonerr)
+ PG_RE_THROW();
- PG_RE_THROW();
+ /* Prepare to disable the subscription */
+ WorkerErrorRecovery();
+ error_recovery_done = true;
}
PG_END_TRY();
+ if (error_recovery_done)
+ DisableSubscriptionOnError();
+
proc_exit(0);
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 10a86f9..42dd045 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4450,6 +4450,7 @@ getSubscriptions(Archive *fout)
int i_rolname;
int i_substream;
int i_subtwophasestate;
+ int i_subdisableonerr;
int i_subconninfo;
int i_subslotname;
int i_subsynccommit;
@@ -4498,12 +4499,18 @@ getSubscriptions(Archive *fout)
appendPQExpBufferStr(query, " false AS substream,\n");
if (fout->remoteVersion >= 150000)
- appendPQExpBufferStr(query, " s.subtwophasestate\n");
+ appendPQExpBufferStr(query, " s.subtwophasestate,\n");
else
appendPQExpBuffer(query,
- " '%c' AS subtwophasestate\n",
+ " '%c' AS subtwophasestate,\n",
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBuffer(query, " s.subdisableonerr\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdisableonerr\n");
+
appendPQExpBufferStr(query,
"FROM pg_subscription s\n"
"WHERE s.subdbid = (SELECT oid FROM pg_database\n"
@@ -4524,6 +4531,7 @@ getSubscriptions(Archive *fout)
i_subbinary = PQfnumber(res, "subbinary");
i_substream = PQfnumber(res, "substream");
i_subtwophasestate = PQfnumber(res, "subtwophasestate");
+ i_subdisableonerr = PQfnumber(res, "subdisableonerr");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4551,6 +4559,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_substream));
subinfo[i].subtwophasestate =
pg_strdup(PQgetvalue(res, i, i_subtwophasestate));
+ subinfo[i].subdisableonerr =
+ pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
if (strlen(subinfo[i].rolname) == 0)
pg_log_warning("owner of subscription \"%s\" appears to be invalid",
@@ -4620,6 +4630,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0)
appendPQExpBufferStr(query, ", two_phase = on");
+ if (strcmp(subinfo->subdisableonerr, "f") != 0)
+ appendPQExpBufferStr(query, ", disable_on_error = on");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 6dccb4b..1cb5cd2 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -658,6 +658,7 @@ typedef struct _SubscriptionInfo
char *subbinary;
char *substream;
char *subtwophasestate;
+ char *subdisableonerr;
char *subsynccommit;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 72d8547..a273648 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6508,7 +6508,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false};
+ false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6542,11 +6542,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Binary"),
gettext_noop("Streaming"));
- /* Two_phase is only supported in v15 and higher */
+ /* Two_phase and disable_on_error is only supported in v15 and higher */
if (pset.sversion >= 150000)
appendPQExpBuffer(&buf,
- ", subtwophasestate AS \"%s\"\n",
- gettext_noop("Two phase commit"));
+ ", subtwophasestate AS \"%s\"\n"
+ ", subdisableonerr AS \"%s\"\n",
+ gettext_noop("Two phase commit"),
+ gettext_noop("Disable On Error"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 2f412ca..5497290 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1681,7 +1681,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit", "disable_on_error");
/* ALTER SUBSCRIPTION <name> SET PUBLICATION */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "PUBLICATION"))
{
@@ -2939,7 +2939,7 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
"enabled", "slot_name", "streaming",
- "synchronous_commit", "two_phase");
+ "synchronous_commit", "two_phase", "disable_on_error");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 2106149..284a90d 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -67,6 +67,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
char subtwophasestate; /* Stream two-phase transactions */
+ bool subdisableonerr; /* True if apply errors should disable the
+ * subscription upon error */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -103,6 +106,7 @@ typedef struct Subscription
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
char twophasestate; /* Allow streaming two-phase transactions */
+ bool disableonerr; /* Whether errors automatically disable */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 80aae83..db2d9ee 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -94,10 +94,10 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
@@ -129,10 +129,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -165,19 +165,19 @@ ERROR: binary requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -188,19 +188,19 @@ ERROR: streaming requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication already exists
@@ -215,10 +215,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication used more then once
@@ -233,10 +233,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -270,10 +270,10 @@ ERROR: two_phase requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | f | off | dbname=regress_doesnotexist
(1 row)
--fail - alter of two_phase option not supported.
@@ -282,10 +282,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -294,10 +294,33 @@ DROP SUBSCRIPTION regress_testsub;
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+ERROR: disable_on_error requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable On Error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | t | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index bd0f4af..dbac8e2 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -228,6 +228,20 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+
+\dRs+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/t/027_disable_on_error.pl b/src/test/subscription/t/027_disable_on_error.pl
new file mode 100644
index 0000000..1104a50
--- /dev/null
+++ b/src/test/subscription/t/027_disable_on_error.pl
@@ -0,0 +1,203 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+
+# Wait for the named subscriptions to catch up or to be disabled.
+sub wait_for_subscriptions
+{
+ my ($node_name, $dbname, @subscriptions) = @_;
+
+ # Unique-ify the subscriptions passed by the caller
+ my %unique = map { $_ => 1 } @subscriptions;
+ my @unique = sort keys %unique;
+ my $unique_count = scalar(@unique);
+
+ # Construct a SQL list from the unique subscription names
+ my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
+ my $sublist = join(', ', map { "'$_'" } @escaped);
+
+ my $polling_sql = qq(
+ SELECT COUNT(1) = $unique_count FROM
+ (SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ LEFT JOIN pg_catalog.pg_subscription_rel sr
+ ON sr.srsubid = s.oid
+ WHERE (sr IS NULL OR sr.srsubstate IN ('s', 'r'))
+ AND s.subname IN ($sublist)
+ AND s.subenabled IS TRUE
+ UNION
+ SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ WHERE s.subname IN ($sublist)
+ AND s.subenabled IS FALSE
+ ) AS synced_or_disabled
+ );
+ return $node_name->poll_query_until($dbname, $polling_sql);
+}
+
+my @schemas = qw(s1 s2);
+my ($schema, $cmd);
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+# Create identical schema, table and index on both the publisher and
+# subscriber
+for $schema (@schemas)
+{
+ $cmd = qq(
+CREATE SCHEMA $schema;
+CREATE TABLE $schema.tbl (i INT);
+ALTER TABLE $schema.tbl REPLICA IDENTITY FULL;
+CREATE INDEX ${schema}_tbl_idx ON $schema.tbl(i));
+ $node_publisher->safe_psql('postgres', $cmd);
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Create non-unique data in both schemas on the publisher.
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (1), (1), (1));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Create an additional unique index in schema s1 on the subscriber only. When
+# we create subscriptions, below, this should cause subscription "s1" on the
+# subscriber to fail during initial synchronization and to get automatically
+# disabled.
+$cmd = qq(CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Create publications and subscriptions linking the schemas on
+# the publisher with those on the subscriber. This tests that the
+# uniqueness violations cause subscription "s1" to fail during
+# initial synchronization.
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+for $schema (@schemas)
+{
+ # Create the publication for this table
+ $cmd = qq(
+CREATE PUBLICATION $schema FOR TABLE $schema.tbl);
+ $node_publisher->safe_psql('postgres', $cmd);
+
+ # Create the subscription for this table
+ $cmd = qq(
+CREATE SUBSCRIPTION $schema
+ CONNECTION '$publisher_connstr'
+ PUBLICATION $schema
+ WITH (disable_on_error = true));
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Wait for the initial subscription synchronizations to finish or fail.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT i, COUNT(*) FROM s2.tbl GROUP BY i);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "1|3", "subscription s2 replicated initial data");
+
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (2));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that the
+# problems encountered by subscription "s1" do not cause subscription "s2" to
+# get stuck.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should still be disabled and have replicated no data
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 still disabled");
+
+# Subscription "s2" should still be enabled and have replicated all changes
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s2 replicated data");
+
+# Drop the unique index on "s1" which caused the subscription to be disabled
+$cmd = qq(DROP INDEX s1.s1_tbl_unique);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Re-enable the subscription "s1"
+$cmd = q(ALTER SUBSCRIPTION s1 ENABLE);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Wait for the data to replicate
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check that we have the new data in s1.tbl
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s1.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s1 replicated data");
+
+# Delete the data from the subscriber only, and recreate the unique index
+$cmd = q(
+DELETE FROM s1.tbl;
+CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Add more non-unique data to the publisher
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (3), (3), (3));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that
+# uniqueness violations encountered during replication cause s1 to be disabled.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "3|7", "subscription s2 replicated additional data");
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-16 05:32 Greg Nancarrow <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Greg Nancarrow @ 2021-12-16 05:32 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Dec 14, 2021 at 4:34 PM [email protected]
<[email protected]> wrote:
>
> Besides all of those changes, I've removed the obsolete
> comment of DisableSubscriptionOnError in v12.
>
I have a few minor comments, otherwise the patch LGTM at this point:
doc/src/sgml/catalogs.sgml
(1)
Current comment says:
+ If true, the subscription will be disabled when subscription's
+ worker detects any errors
However, in create_subscription.sgml, it says "disabled if any errors
are detected by subscription workers ..."
For consistency, I think it should be:
+ If true, the subscription will be disabled when subscription
+ workers detect any errors
src/bin/psql/describe.c
(2)
I think that:
+ gettext_noop("Disable On Error"));
should be:
+ gettext_noop("Disable on error"));
for consistency with the uppercase/lowercase usage on other similar entries?
(e.g. "Two phase commit")
src/include/catalog/pg_subscription.h
(3)
+ bool subdisableonerr; /* True if apply errors should disable the
+ * subscription upon error */
The comment should just say "True if occurrence of apply errors should
disable the subscription"
Regards,
Greg Nancarrow
Fujitsu Australia
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-16 12:51 [email protected] <[email protected]>
parent: Greg Nancarrow <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: [email protected] @ 2021-12-16 12:51 UTC (permalink / raw)
To: 'Greg Nancarrow' <[email protected]>; +Cc: vignesh C <[email protected]>; Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thursday, December 16, 2021 2:32 PM Greg Nancarrow <[email protected]> wrote:
> On Tue, Dec 14, 2021 at 4:34 PM [email protected]
> <[email protected]> wrote:
> >
> > Besides all of those changes, I've removed the obsolete comment of
> > DisableSubscriptionOnError in v12.
> >
>
> I have a few minor comments, otherwise the patch LGTM at this point:
Thank you for your review !
> doc/src/sgml/catalogs.sgml
> (1)
> Current comment says:
>
> + If true, the subscription will be disabled when subscription's
> + worker detects any errors
>
> However, in create_subscription.sgml, it says "disabled if any errors are
> detected by subscription workers ..."
>
> For consistency, I think it should be:
>
> + If true, the subscription will be disabled when subscription
> + workers detect any errors
Okay. Fixed.
> src/bin/psql/describe.c
> (2)
> I think that:
>
> + gettext_noop("Disable On Error"));
>
> should be:
>
> + gettext_noop("Disable on error"));
>
> for consistency with the uppercase/lowercase usage on other similar entries?
> (e.g. "Two phase commit")
Agreed. Fixed.
> src/include/catalog/pg_subscription.h
> (3)
>
> + bool subdisableonerr; /* True if apply errors should disable the
> + * subscription upon error */
>
> The comment should just say "True if occurrence of apply errors should disable
> the subscription"
Fixed.
Attached the updated patch v14.
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v14-0001-Optionally-disable-subscriptions-on-error.patch (49.0K, ../../TYCPR01MB837344D0306FFCAAE280F498ED779@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v14-0001-Optionally-disable-subscriptions-on-error.patch)
download | inline diff:
From efa16846107279586e84aa31665783217c8db83f Mon Sep 17 00:00:00 2001
From: Takamichi Osumi <[email protected]>
Date: Thu, 16 Dec 2021 12:28:12 +0000
Subject: [PATCH v14] Optionally disable subscriptions on error
Logical replication apply workers for a subscription can easily get
stuck in an infinite loop of attempting to apply a change,
triggering an error (such as a constraint violation), exiting with
an error written to the subscription worker log, and restarting.
To partially remedy the situation, adding a new
subscription_parameter named 'disable_on_error'. To be consistent
with old behavior, the parameter defaults to false. When true, both
the table sync worker and apply worker catch any errors thrown and
disable the subscription in order to break the loop. The error is
still also written to the logs.
Require to bump catalog version.
Proposed and written originally by Mark Dilger
Taken over by Osumi Takamichi, Greg Nancarrow
Reviewed by Greg Nancarrow, Vignesh C, Amit Kapila
Discussion : https://www.postgresql.org/message-id/DB35438F-9356-4841-89A0-412709EBD3AB%40enterprisedb.com
---
doc/src/sgml/catalogs.sgml | 10 ++
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 ++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 27 +++-
src/backend/replication/logical/worker.c | 139 +++++++++++++++-
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 10 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 4 +
src/test/regress/expected/subscription.out | 119 ++++++++------
src/test/regress/sql/subscription.sql | 14 ++
src/test/subscription/t/027_disable_on_error.pl | 203 ++++++++++++++++++++++++
15 files changed, 499 insertions(+), 69 deletions(-)
create mode 100644 src/test/subscription/t/027_disable_on_error.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 03e2537..f89257d 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7725,6 +7725,16 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<row>
<entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdisableonerr</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription will be disabled when subscription
+ workers detect any errors
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
</para>
<para>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 0b027cc..3109ee9 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -201,8 +201,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
information. The parameters that can be altered
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
- <literal>binary</literal>, and
- <literal>streaming</literal>.
+ <literal>binary</literal>,<literal>streaming</literal>, and
+ <literal>disable_on_error</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 990a41f..9ca0bb9 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -142,6 +142,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</varlistentry>
<varlistentry>
+ <term><literal>disable_on_error</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription should be automatically disabled
+ if any errors are detected by subscription workers during data
+ replication from the publisher. The default is
+ <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>enabled</literal> (<type>boolean</type>)</term>
<listitem>
<para>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 25021e2..9b416dd 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -69,6 +69,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->binary = subform->subbinary;
sub->stream = subform->substream;
sub->twophasestate = subform->subtwophasestate;
+ sub->disableonerr = subform->subdisableonerr;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 61b515c..9d0ed61 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1259,7 +1259,8 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subname, subowner, subenabled, subbinary,
- substream, subtwophasestate, subslotname, subsynccommit, subpublications)
+ substream, subtwophasestate, subdisableonerr, subslotname,
+ subsynccommit, subpublications)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_workers AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 2b65808..cebc51a 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -61,6 +61,7 @@
#define SUBOPT_BINARY 0x00000080
#define SUBOPT_STREAMING 0x00000100
#define SUBOPT_TWOPHASE_COMMIT 0x00000200
+#define SUBOPT_DISABLE_ON_ERR 0x00000400
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -82,6 +83,7 @@ typedef struct SubOpts
bool binary;
bool streaming;
bool twophase;
+ bool disableonerr;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -130,6 +132,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->streaming = false;
if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT))
opts->twophase = false;
+ if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
+ opts->disableonerr = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -249,6 +253,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_TWOPHASE_COMMIT;
opts->twophase = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR) &&
+ strcmp(defel->defname, "disable_on_error") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DISABLE_ON_ERR))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DISABLE_ON_ERR;
+ opts->disableonerr = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -390,7 +403,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
supported_opts = (SUBOPT_CONNECT | SUBOPT_ENABLED | SUBOPT_CREATE_SLOT |
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT);
+ SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
+ SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -464,6 +478,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
CharGetDatum(opts.twophase ?
LOGICALREP_TWOPHASE_STATE_PENDING :
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -864,7 +879,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
{
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING);
+ SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -913,6 +928,14 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_substream - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DISABLE_ON_ERR))
+ {
+ values[Anum_pg_subscription_subdisableonerr - 1]
+ = BoolGetDatum(opts.disableonerr);
+ replaces[Anum_pg_subscription_subdisableonerr - 1]
+ = true;
+ }
+
update_tuple = true;
break;
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 2e79302..dc81f85 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -136,6 +136,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/partition.h"
#include "catalog/pg_inherits.h"
@@ -2755,6 +2756,92 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/*
+ * Worker error recovery processing, in preparation for disabling the
+ * subscription.
+ */
+static void
+WorkerErrorRecovery(void)
+{
+ /* Notify the subscription will be no longer valid */
+ ereport(LOG,
+ errmsg("logical replication subscription \"%s\" will be disabled due to an error",
+ MySubscription->name));
+ /* Emit the error */
+ EmitErrorReport();
+ /* Abort any active transaction */
+ AbortOutOfAnyTransaction();
+ /* Reset the ErrorContext */
+ FlushErrorState();
+}
+
+/*
+ * Disable the current subscription.
+ */
+static void
+DisableSubscriptionOnError(void)
+{
+ Relation rel;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+ HeapTuple tup;
+ Form_pg_subscription subform;
+
+ /* Disable the subscription in a fresh transaction */
+ StartTransactionCommand();
+
+ /* Look up our subscription in the catalogs */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy2(SUBSCRIPTIONNAME, MyDatabaseId,
+ CStringGetDatum(MySubscription->name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("subscription \"%s\" does not exist",
+ MySubscription->name));
+
+ subform = (Form_pg_subscription) GETSTRUCT(tup);
+
+ /*
+ * We would not be here unless this subscription's disableonerr field was
+ * true when our worker began applying changes, but check whether that
+ * field has changed in the interim.
+ */
+ if (!subform->subdisableonerr)
+ {
+ /*
+ * Disabling the subscription has been done already. No need of
+ * additional work.
+ */
+ heap_freetuple(tup);
+ table_close(rel, RowExclusiveLock);
+ CommitTransactionCommand();
+ return;
+ }
+
+ LockSharedObject(SubscriptionRelationId, subform->oid, 0, AccessExclusiveLock);
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Set the subscription to disabled, and note the reason. */
+ values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(false);
+ replaces[Anum_pg_subscription_subenabled - 1] = true;
+
+ /* Update the catalog */
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+ heap_freetuple(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ CommitTransactionCommand();
+}
+
+/*
* Send a Standby Status Update message to server.
*
* 'recvpos' is the latest LSN we've received data to, force is set if we need
@@ -3339,6 +3426,7 @@ ApplyWorkerMain(Datum main_arg)
char *myslotname;
WalRcvStreamOptions options;
int server_version;
+ bool error_recovery_done = false;
/* Attach to slot */
logicalrep_worker_attach(worker_slot);
@@ -3453,15 +3541,33 @@ ApplyWorkerMain(Datum main_arg)
0, /* message type */
InvalidTransactionId,
errdata->message);
- MemoryContextSwitchTo(ecxt);
- PG_RE_THROW();
+
+ if (MySubscription->disableonerr)
+ {
+ WorkerErrorRecovery();
+ error_recovery_done = true;
+ }
+ else
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
- /* allocate slot name in long-lived context */
- myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
+ /* If we caught an error above, disable the subscription */
+ if (error_recovery_done)
+ {
+ DisableSubscriptionOnError();
+ proc_exit(0);
+ }
+ else
+ {
+ /* allocate slot name in long-lived context */
+ myslotname = MemoryContextStrdup(ApplyContext, syncslotname);
- pfree(syncslotname);
+ pfree(syncslotname);
+ }
}
else
{
@@ -3580,10 +3686,11 @@ ApplyWorkerMain(Datum main_arg)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(cctx);
+
/* report the apply error */
if (apply_error_callback_arg.command != 0)
{
- MemoryContext ecxt = MemoryContextSwitchTo(cctx);
ErrorData *errdata = CopyErrorData();
pgstat_report_subworker_error(MyLogicalRepWorker->subid,
@@ -3594,13 +3701,29 @@ ApplyWorkerMain(Datum main_arg)
apply_error_callback_arg.command,
apply_error_callback_arg.remote_xid,
errdata->message);
- MemoryContextSwitchTo(ecxt);
+
+ if (!MySubscription->disableonerr)
+ {
+ /*
+ * Some work in error recovery work is done. Switch to the old
+ * memory context and rethrow.
+ */
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
+ else if (!MySubscription->disableonerr)
+ PG_RE_THROW();
- PG_RE_THROW();
+ /* Prepare to disable the subscription */
+ WorkerErrorRecovery();
+ error_recovery_done = true;
}
PG_END_TRY();
+ if (error_recovery_done)
+ DisableSubscriptionOnError();
+
proc_exit(0);
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 326441a..1afe9ce 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4226,6 +4226,7 @@ getSubscriptions(Archive *fout)
int i_rolname;
int i_substream;
int i_subtwophasestate;
+ int i_subdisableonerr;
int i_subconninfo;
int i_subslotname;
int i_subsynccommit;
@@ -4274,12 +4275,18 @@ getSubscriptions(Archive *fout)
appendPQExpBufferStr(query, " false AS substream,\n");
if (fout->remoteVersion >= 150000)
- appendPQExpBufferStr(query, " s.subtwophasestate\n");
+ appendPQExpBufferStr(query, " s.subtwophasestate,\n");
else
appendPQExpBuffer(query,
- " '%c' AS subtwophasestate\n",
+ " '%c' AS subtwophasestate,\n",
LOGICALREP_TWOPHASE_STATE_DISABLED);
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBuffer(query, " s.subdisableonerr\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdisableonerr\n");
+
appendPQExpBufferStr(query,
"FROM pg_subscription s\n"
"WHERE s.subdbid = (SELECT oid FROM pg_database\n"
@@ -4300,6 +4307,7 @@ getSubscriptions(Archive *fout)
i_subbinary = PQfnumber(res, "subbinary");
i_substream = PQfnumber(res, "substream");
i_subtwophasestate = PQfnumber(res, "subtwophasestate");
+ i_subdisableonerr = PQfnumber(res, "subdisableonerr");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4327,6 +4335,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_substream));
subinfo[i].subtwophasestate =
pg_strdup(PQgetvalue(res, i, i_subtwophasestate));
+ subinfo[i].subdisableonerr =
+ pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
if (strlen(subinfo[i].rolname) == 0)
pg_log_warning("owner of subscription \"%s\" appears to be invalid",
@@ -4396,6 +4406,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0)
appendPQExpBufferStr(query, ", two_phase = on");
+ if (strcmp(subinfo->subdisableonerr, "f") != 0)
+ appendPQExpBufferStr(query, ", disable_on_error = on");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 6dccb4b..1cb5cd2 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -658,6 +658,7 @@ typedef struct _SubscriptionInfo
char *subbinary;
char *substream;
char *subtwophasestate;
+ char *subdisableonerr;
char *subsynccommit;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 72d8547..4690cb7 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6508,7 +6508,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false};
+ false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6542,11 +6542,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Binary"),
gettext_noop("Streaming"));
- /* Two_phase is only supported in v15 and higher */
+ /* Two_phase and disable_on_error is only supported in v15 and higher */
if (pset.sversion >= 150000)
appendPQExpBuffer(&buf,
- ", subtwophasestate AS \"%s\"\n",
- gettext_noop("Two phase commit"));
+ ", subtwophasestate AS \"%s\"\n"
+ ", subdisableonerr AS \"%s\"\n",
+ gettext_noop("Two phase commit"),
+ gettext_noop("Disable on error"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 2f412ca..5497290 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1681,7 +1681,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit", "disable_on_error");
/* ALTER SUBSCRIPTION <name> SET PUBLICATION */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "PUBLICATION"))
{
@@ -2939,7 +2939,7 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
"enabled", "slot_name", "streaming",
- "synchronous_commit", "two_phase");
+ "synchronous_commit", "two_phase", "disable_on_error");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 2106149..7b76842 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -67,6 +67,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
char subtwophasestate; /* Stream two-phase transactions */
+ bool subdisableonerr; /* True if occurrence of apply errors
+ * should disable the subscription */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -103,6 +106,7 @@ typedef struct Subscription
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
char twophasestate; /* Allow streaming two-phase transactions */
+ bool disableonerr; /* Whether errors automatically disable */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 80aae83..ad8003f 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -94,10 +94,10 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
@@ -129,10 +129,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | f | d | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -165,19 +165,19 @@ ERROR: binary requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | f | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -188,19 +188,19 @@ ERROR: streaming requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication already exists
@@ -215,10 +215,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
-- fail - publication used more then once
@@ -233,10 +233,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -270,10 +270,10 @@ ERROR: two_phase requires a Boolean value
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | p | f | off | dbname=regress_doesnotexist
(1 row)
--fail - alter of two_phase option not supported.
@@ -282,10 +282,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -294,10 +294,33 @@ DROP SUBSCRIPTION regress_testsub;
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true, two_phase = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------+-----------+------------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | t | p | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+ERROR: disable_on_error requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | f | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two phase commit | Disable on error | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | f | d | t | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index bd0f4af..dbac8e2 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -228,6 +228,20 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - disable_on_error must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
+
+\dRs+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/t/027_disable_on_error.pl b/src/test/subscription/t/027_disable_on_error.pl
new file mode 100644
index 0000000..1104a50
--- /dev/null
+++ b/src/test/subscription/t/027_disable_on_error.pl
@@ -0,0 +1,203 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test of logical replication subscription self-disabling feature
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+
+# Wait for the named subscriptions to catch up or to be disabled.
+sub wait_for_subscriptions
+{
+ my ($node_name, $dbname, @subscriptions) = @_;
+
+ # Unique-ify the subscriptions passed by the caller
+ my %unique = map { $_ => 1 } @subscriptions;
+ my @unique = sort keys %unique;
+ my $unique_count = scalar(@unique);
+
+ # Construct a SQL list from the unique subscription names
+ my @escaped = map { s/'/''/g; s/\\/\\\\/g; $_ } @unique;
+ my $sublist = join(', ', map { "'$_'" } @escaped);
+
+ my $polling_sql = qq(
+ SELECT COUNT(1) = $unique_count FROM
+ (SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ LEFT JOIN pg_catalog.pg_subscription_rel sr
+ ON sr.srsubid = s.oid
+ WHERE (sr IS NULL OR sr.srsubstate IN ('s', 'r'))
+ AND s.subname IN ($sublist)
+ AND s.subenabled IS TRUE
+ UNION
+ SELECT s.oid
+ FROM pg_catalog.pg_subscription s
+ WHERE s.subname IN ($sublist)
+ AND s.subenabled IS FALSE
+ ) AS synced_or_disabled
+ );
+ return $node_name->poll_query_until($dbname, $polling_sql);
+}
+
+my @schemas = qw(s1 s2);
+my ($schema, $cmd);
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+# Create identical schema, table and index on both the publisher and
+# subscriber
+for $schema (@schemas)
+{
+ $cmd = qq(
+CREATE SCHEMA $schema;
+CREATE TABLE $schema.tbl (i INT);
+ALTER TABLE $schema.tbl REPLICA IDENTITY FULL;
+CREATE INDEX ${schema}_tbl_idx ON $schema.tbl(i));
+ $node_publisher->safe_psql('postgres', $cmd);
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Create non-unique data in both schemas on the publisher.
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (1), (1), (1));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Create an additional unique index in schema s1 on the subscriber only. When
+# we create subscriptions, below, this should cause subscription "s1" on the
+# subscriber to fail during initial synchronization and to get automatically
+# disabled.
+$cmd = qq(CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Create publications and subscriptions linking the schemas on
+# the publisher with those on the subscriber. This tests that the
+# uniqueness violations cause subscription "s1" to fail during
+# initial synchronization.
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+for $schema (@schemas)
+{
+ # Create the publication for this table
+ $cmd = qq(
+CREATE PUBLICATION $schema FOR TABLE $schema.tbl);
+ $node_publisher->safe_psql('postgres', $cmd);
+
+ # Create the subscription for this table
+ $cmd = qq(
+CREATE SUBSCRIPTION $schema
+ CONNECTION '$publisher_connstr'
+ PUBLICATION $schema
+ WITH (disable_on_error = true));
+ $node_subscriber->safe_psql('postgres', $cmd);
+}
+
+# Wait for the initial subscription synchronizations to finish or fail.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT i, COUNT(*) FROM s2.tbl GROUP BY i);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "1|3", "subscription s2 replicated initial data");
+
+# Enter unique data for both schemas on the publisher. This should succeed on
+# the publisher node, and not cause any additional problems on the subscriber
+# side either, though disabled subscription "s1" should not replicate anything.
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (2));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that the
+# problems encountered by subscription "s1" do not cause subscription "s2" to
+# get stuck.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should still be disabled and have replicated no data
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 still disabled");
+
+# Subscription "s2" should still be enabled and have replicated all changes
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s2 replicated data");
+
+# Drop the unique index on "s1" which caused the subscription to be disabled
+$cmd = qq(DROP INDEX s1.s1_tbl_unique);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Re-enable the subscription "s1"
+$cmd = q(ALTER SUBSCRIPTION s1 ENABLE);
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Wait for the data to replicate
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check that we have the new data in s1.tbl
+$cmd = q(SELECT MAX(i), COUNT(*) FROM s1.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "2|4", "subscription s1 replicated data");
+
+# Delete the data from the subscriber only, and recreate the unique index
+$cmd = q(
+DELETE FROM s1.tbl;
+CREATE UNIQUE INDEX s1_tbl_unique ON s1.tbl (i));
+$node_subscriber->safe_psql('postgres', $cmd);
+
+# Add more non-unique data to the publisher
+for $schema (@schemas)
+{
+ $cmd = qq(INSERT INTO $schema.tbl (i) VALUES (3), (3), (3));
+ $node_publisher->safe_psql('postgres', $cmd);
+}
+
+# Wait for the data to replicate for the subscriptions. This tests that
+# uniqueness violations encountered during replication cause s1 to be disabled.
+wait_for_subscriptions($node_subscriber, 'postgres', @schemas)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Subscription "s1" should have disabled itself due to error.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's1');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "f", "subscription s1 no longer enabled");
+
+# Subscription "s2" should have copied the initial data without incident.
+$cmd = qq(
+SELECT subenabled FROM pg_catalog.pg_subscription WHERE subname = 's2');
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "t", "subscription s2 still enabled");
+$cmd = qq(SELECT MAX(i), COUNT(*) FROM s2.tbl);
+is($node_subscriber->safe_psql('postgres', $cmd),
+ "3|7", "subscription s2 replicated additional data");
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-21 14:17 [email protected] <[email protected]>
parent: [email protected] <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: [email protected] @ 2021-12-21 14:17 UTC (permalink / raw)
To: 'Greg Nancarrow' <[email protected]>; +Cc: vignesh C <[email protected]>; Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thursday, December 16, 2021 9:51 PM I wrote:
> Attached the updated patch v14.
FYI, I've conducted a test of disable_on_error flag using
pg_upgrade. I prepared PG14 and HEAD applied with disable_on_error patch.
Then, I setup a logical replication pair of the publisher and the subscriber by 14
and executed pg_upgrade for both the publisher and the subscriber individually.
After the updation, on the subscriber, I've confirmed the disable_on_error is false
via both pg_subscription and \dRs+, as expected.
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-22 10:24 [email protected] <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 0 replies; 74+ messages in thread
From: [email protected] @ 2021-12-22 10:24 UTC (permalink / raw)
To: 'Greg Nancarrow' <[email protected]>; +Cc: vignesh C <[email protected]>; Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tuesday, December 21, 2021 11:18 PM I wrote:
> On Thursday, December 16, 2021 9:51 PM I wrote:
> > Attached the updated patch v14.
> FYI, I've conducted a test of disable_on_error flag using pg_upgrade. I
> prepared PG14 and HEAD applied with disable_on_error patch.
> Then, I setup a logical replication pair of the publisher and the subscriber by 14
> and executed pg_upgrade for both the publisher and the subscriber
> individually.
>
> After the updation, on the subscriber, I've confirmed the disable_on_error is
> false via both pg_subscription and \dRs+, as expected.
Additionally, I've tested the new TAP test in a tight loop
that executed 027_disable_on_error.pl 100 times sequentially.
There was no failure, which means
any timing issue should not exist in the test.
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 74+ messages in thread
* RE: Optionally automatically disable logical replication subscriptions on error
@ 2021-12-28 02:52 [email protected] <[email protected]>
parent: [email protected] <[email protected]>
1 sibling, 0 replies; 74+ messages in thread
From: [email protected] @ 2021-12-28 02:52 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; 'Greg Nancarrow' <[email protected]>; Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Mark Dilger <[email protected]>; Smith, Peter <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thursday, December 16, 2021 8:51 PM [email protected] <[email protected]> wrote:
> Attached the updated patch v14.
A comment to the timing of printing a log:
After the log[1] was printed, I altered subscription's option
(DISABLE_ON_ERROR) from true to false before invoking DisableSubscriptionOnError
to disable subscription. Subscription was not disabled.
[1] "LOG: logical replication subscription "sub1" will be disabled due to an error"
I found this log is printed in function WorkerErrorRecovery:
+ ereport(LOG,
+ errmsg("logical replication subscription \"%s\" will be disabled due to an error",
+ MySubscription->name));
This log is printed here, but in DisableSubscriptionOnError, there is a check to
confirm subscription's disableonerr field. If disableonerr is found changed from
true to false in DisableSubscriptionOnError, subscription will not be disabled.
In this case, "disable subscription" is printed, but subscription will not be
disabled actually.
I think it is a little confused to user, so what about moving this message after
the check which is mentioned above in DisableSubscriptionOnError?
Regards,
Wang wei
^ permalink raw reply [nested|flat] 74+ messages in thread
end of thread, other threads:[~2021-12-28 02:52 UTC | newest]
Thread overview: 74+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-03-10 23:20 [PATCH] pgbench: increase the maximum number of variables/arguments Dagfinn Ilmari Mannsåker <[email protected]>
2021-06-17 20:18 Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-06-18 04:47 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-06-18 19:36 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-06-19 10:17 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-06-19 14:44 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-06-19 16:21 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-06-21 02:17 ` Re: Optionally automatically disable logical replication subscriptions on error Masahiko Sawada <[email protected]>
2021-06-21 02:26 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-06-21 03:09 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-06-21 03:50 ` Re: Optionally automatically disable logical replication subscriptions on error Masahiko Sawada <[email protected]>
2021-06-21 05:26 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-06-21 04:54 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-06-21 05:11 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-06-21 05:16 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-06-21 05:25 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-06-21 05:49 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-06-21 10:47 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-06-22 00:57 ` Re: Optionally automatically disable logical replication subscriptions on error Peter Smith <[email protected]>
2021-06-22 02:29 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-06-22 02:35 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-06-22 02:44 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-06-22 02:49 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-06-22 02:42 ` Re: Optionally automatically disable logical replication subscriptions on error Masahiko Sawada <[email protected]>
2021-06-23 02:53 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-06-21 05:12 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-06-28 04:47 ` Re: Optionally automatically disable logical replication subscriptions on error Masahiko Sawada <[email protected]>
2021-11-02 10:42 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-11-08 13:14 ` Re: Optionally automatically disable logical replication subscriptions on error vignesh C <[email protected]>
2021-11-10 01:26 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-11-10 04:22 ` Re: Optionally automatically disable logical replication subscriptions on error Greg Nancarrow <[email protected]>
2021-11-10 04:31 ` Re: Optionally automatically disable logical replication subscriptions on error Greg Nancarrow <[email protected]>
2021-11-11 09:20 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-11-12 04:08 ` Re: Optionally automatically disable logical replication subscriptions on error vignesh C <[email protected]>
2021-11-16 07:53 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-11-18 05:07 ` Re: Optionally automatically disable logical replication subscriptions on error Greg Nancarrow <[email protected]>
2021-11-18 07:22 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-11-22 06:52 ` Re: Optionally automatically disable logical replication subscriptions on error vignesh C <[email protected]>
2021-11-26 14:36 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-11-29 05:37 ` Re: Optionally automatically disable logical replication subscriptions on error vignesh C <[email protected]>
2021-11-30 12:13 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-11-30 04:09 ` Re: Optionally automatically disable logical replication subscriptions on error Greg Nancarrow <[email protected]>
2021-11-30 12:04 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-12-01 06:01 ` Re: Optionally automatically disable logical replication subscriptions on error vignesh C <[email protected]>
2021-12-01 12:25 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-12-01 13:16 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-12-02 01:05 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-12-02 02:42 ` Re: Optionally automatically disable logical replication subscriptions on error Greg Nancarrow <[email protected]>
2021-12-02 04:48 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-12-02 07:40 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-12-03 13:20 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-12-06 04:16 ` Re: Optionally automatically disable logical replication subscriptions on error Greg Nancarrow <[email protected]>
2021-12-06 10:52 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-12-13 09:57 ` Re: Optionally automatically disable logical replication subscriptions on error vignesh C <[email protected]>
2021-12-14 05:34 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-12-16 05:32 ` Re: Optionally automatically disable logical replication subscriptions on error Greg Nancarrow <[email protected]>
2021-12-16 12:51 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-12-21 14:17 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-12-22 10:24 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-12-28 02:52 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-12-06 04:37 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-12-06 06:56 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-12-06 16:06 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-12-07 00:22 ` Re: Optionally automatically disable logical replication subscriptions on error Greg Nancarrow <[email protected]>
2021-12-08 13:10 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-12-08 15:52 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-12-09 04:09 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-12-09 16:44 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
2021-12-06 10:59 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-11-12 04:48 ` Re: Optionally automatically disable logical replication subscriptions on error Greg Nancarrow <[email protected]>
2021-11-16 07:59 ` RE: Optionally automatically disable logical replication subscriptions on error [email protected] <[email protected]>
2021-06-21 02:53 ` Re: Optionally automatically disable logical replication subscriptions on error Amit Kapila <[email protected]>
2021-06-18 06:34 ` Re: Optionally automatically disable logical replication subscriptions on error Peter Smith <[email protected]>
2021-06-19 00:03 ` Re: Optionally automatically disable logical replication subscriptions on error Mark Dilger <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox