public inbox for [email protected]
help / color / mirror / Atom feedRe: Time delayed LR (WAS Re: logical replication restrictions)
40+ messages / 9 participants
[nested] [flat]
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-12 03:03 Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 40+ messages in thread
From: Kyotaro Horiguchi @ 2023-01-12 03:03 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]
At Wed, 11 Jan 2023 12:46:24 +0000, "Hayato Kuroda (Fujitsu)" <[email protected]> wrote in
> them. Which version is better?
Some comments by a quick loock, different from the above.
+ CONNECTION 'host=192.168.1.50 port=5432 user=foo dbname=foodb'
I understand that we (not PG people, but IT people) are supposed to
use in documents a certain set of special addresses that is guaranteed
not to be routed in the field.
> TEST-NET-1 : 192.0.2.0/24
> TEST-NET-2 : 198.51.100.0/24
> TEST-NET-3 : 203.0.113.0/24
(I found 192.83.123.89 in the postgres_fdw doc, but it'd be another issue..)
+ if (strspn(tmp, "-0123456789 ") == strlen(tmp))
Do we need to bother spending another memory block for apparent
non-digits here?
+ errmsg(INT64_FORMAT " ms is outside the valid range for parameter \"%s\"",
We don't use INT64_FORMAT in translatable message strings. Cast then
use %lld instead.
This message looks unfriendly as it doesn't suggest the valid range,
and it shows the input value in a different unit from what was in the
input. A I think we can spell it as "\"%s\" is outside the valid range
for subsciription parameter \"%s\" (0 .. <INT_MAX> in millisecond)"
+ int64 min_apply_delay;
..
+ if (ms < 0 || ms > INT_MAX)
Why is the variable wider than required?
+ errmsg("%s and %s are mutually exclusive options",
+ "min_apply_delay > 0", "streaming = parallel"));
Mmm. Couldn't we refuse 0 as min_apply_delay?
+ sub->minapplydelay > 0)
...
+ if (opts.min_apply_delay > 0 &&
Is there any reason for the differenciation?
+ errmsg("cannot set %s for subscription with %s",
+ "streaming = parallel", "min_apply_delay > 0"));
I think that this shoud be more like human-speking. Say, "cannot
enable min_apply_delay for subscription in parallel streaming mode" or
something.. The same is applicable to the nearby message.
+static void maybe_delay_apply(TimestampTz ts);
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn, TimestampTz ts)
"ts" looks too generic. Couldn't it be more specific?
We need a explanation for the parameter in the function comment.
+ if (!am_parallel_apply_worker())
+ {
+ Assert(ts > 0);
+ maybe_delay_apply(ts);
It seems to me better that the if condition and assertion are checked
inside maybe_delay_apply().
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-12 15:39 Takamichi Osumi (Fujitsu) <[email protected]>
parent: Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-12 15:39 UTC (permalink / raw)
To: 'Kyotaro Horiguchi' <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>
On Thursday, January 12, 2023 12:04 PM Kyotaro Horiguchi <[email protected]> wrote:
> At Wed, 11 Jan 2023 12:46:24 +0000, "Hayato Kuroda (Fujitsu)"
> <[email protected]> wrote in
> > them. Which version is better?
>
>
> Some comments by a quick loock, different from the above.
Horiguchi-san, thanks for your review !
> + CONNECTION 'host=192.168.1.50 port=5432 user=foo
> dbname=foodb'
>
> I understand that we (not PG people, but IT people) are supposed to use in
> documents a certain set of special addresses that is guaranteed not to be
> routed in the field.
>
> > TEST-NET-1 : 192.0.2.0/24
> > TEST-NET-2 : 198.51.100.0/24
> > TEST-NET-3 : 203.0.113.0/24
>
> (I found 192.83.123.89 in the postgres_fdw doc, but it'd be another issue..)
Fixed. If necessary we can create another thread for this.
> + if (strspn(tmp, "-0123456789 ") == strlen(tmp))
>
> Do we need to bother spending another memory block for apparent non-digits
> here?
Yes. The characters are necessary to handle an issue reported in [1].
The issue happened if the user inputs a negative value,
then the length comparison became different between strspn and strlen
and the input value was recognized as seconds, when
the unit wasn't described. This led to a wrong error message for the user.
Those addition of such characters solve the issue.
> + errmsg(INT64_FORMAT " ms
> is outside the valid range for parameter
> +\"%s\"",
>
> We don't use INT64_FORMAT in translatable message strings. Cast then
> use %lld instead.
Thanks for teaching us. Fixed.
> This message looks unfriendly as it doesn't suggest the valid range, and it
> shows the input value in a different unit from what was in the input. A I think we
> can spell it as "\"%s\" is outside the valid range for subsciription parameter
> \"%s\" (0 .. <INT_MAX> in millisecond)"
Makes sense. I incorporated the valid range with the aligned format of recovery_min_apply_delay.
FYI, the physical replication's GUC doesn't write the unites for the range like below.
I followed and applied this style.
---
LOG: -1 ms is outside the valid range for parameter "recovery_min_apply_delay" (0 .. 2147483647)
FATAL: configuration file "/home/k5user/new/pg/l/make_v15/slave/postgresql.conf" contains errors
---
> + int64 min_apply_delay;
> ..
> + if (ms < 0 || ms > INT_MAX)
>
> Why is the variable wider than required?
You are right. Fixed.
> + errmsg("%s and %s are mutually
> exclusive options",
> + "min_apply_delay > 0",
> "streaming = parallel"));
>
> Mmm. Couldn't we refuse 0 as min_apply_delay?
Sorry, the previous patch's behavior wasn't consistent with this error message.
In the previous patch, if we conducted alter subscription
with stream = parallel and min_apply_delay = 0 (from a positive value) at the same time,
the alter command failed, although this should succeed by this time-delayed feature specification.
We fixed this part accordingly by some more tests in AlterSubscription().
By the way, we should allow users to change min_apply_dealy to 0
whenever they want from different value. Then, we didn't restrict
this kind of operation.
> + sub->minapplydelay > 0)
> ...
> + if (opts.min_apply_delay > 0 &&
>
> Is there any reason for the differenciation?
Yes. The former is the object for an existing subscription configuration.
For example, if we alter subscription with setting streaming = 'parallel'
for a subscription created with min_apply_delay = '1 day', we
need to reject the alter command. The latter is new settings.
> +
> errmsg("cannot set %s for subscription with %s",
> +
> "streaming = parallel", "min_apply_delay > 0"));
>
> I think that this shoud be more like human-speking. Say, "cannot enable
> min_apply_delay for subscription in parallel streaming mode" or something..
> The same is applicable to the nearby message.
Reworded the error messages. Please check.
> +static void maybe_delay_apply(TimestampTz ts);
>
> apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
> - XLogRecPtr lsn)
> + XLogRecPtr lsn, TimestampTz ts)
>
> "ts" looks too generic. Couldn't it be more specific?
> We need a explanation for the parameter in the function comment.
Changed it to finish_ts, since it indicates commit/prepare time.
This terminology should be aligned with finish lsn.
> + if (!am_parallel_apply_worker())
> + {
> + Assert(ts > 0);
> + maybe_delay_apply(ts);
>
> It seems to me better that the if condition and assertion are checked inside
> maybe_delay_apply().
Fixed.
[1] - https://www.postgresql.org/message-id/CALDaNm3Bpzhh60nU-keuGxMPb-OhcqsfpCN3ysfCfCJ-2ShYPA%40mail.gma...
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v15-0001-Time-delayed-logical-replication-subscriber.patch (79.6K, ../../TYCPR01MB83739C6133B50DDA8BAD1601EDFD9@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v15-0001-Time-delayed-logical-replication-subscriber.patch)
download | inline diff:
From aec7ec458e22f1a75e06789a1b36b44191d6b760 Mon Sep 17 00:00:00 2001
From: Takamichi Osumi <[email protected]>
Date: Thu, 12 Jan 2023 15:05:59 +0000
Subject: [PATCH v15] Time-delayed logical replication subscriber
Similar to physical replication, a time-delayed copy of the data for
logical replication is useful for some scenarios (particularly to fix
errors that might cause data loss).
If the subscription sets min_apply_delay parameter, the logical
replication worker will delay the transaction commit for min_apply_delay
milliseconds.
The delay is calculated between the WAL time stamp and the current time
on the subscriber.
The delay occurs only on WAL records for transaction begins. The main
reason is to avoid keeping a transaction open for a long time. Regular
and prepared transactions are covered. Streamed transactions are also
covered.
Using this feature with parallel apply feature is prohibited.
Author: Euler Taveira
Discussion: https://postgr.es/m/CAB-JLwYOYwL=XTyAXKiH5CtM_Vm8KjKh7aaitCKvmCh4rzr5pQ@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/config.sgml | 12 ++
doc/src/sgml/logical-replication.sgml | 7 +
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 59 +++++-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 7 +-
src/backend/commands/subscriptioncmds.c | 92 ++++++++-
.../replication/logical/applyparallelworker.c | 3 +-
src/backend/replication/logical/worker.c | 166 ++++++++++++++--
src/backend/utils/adt/timestamp.c | 29 +++
src/bin/pg_dump/pg_dump.c | 16 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 9 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 3 +
src/include/datatype/timestamp.h | 2 +
src/include/replication/worker_internal.h | 2 +-
src/include/utils/timestamp.h | 2 +
src/test/regress/expected/subscription.out | 185 +++++++++++-------
src/test/regress/sql/subscription.sql | 25 +++
src/test/subscription/meson.build | 1 +
src/test/subscription/t/032_apply_delay.pl | 170 ++++++++++++++++
23 files changed, 708 insertions(+), 102 deletions(-)
create mode 100644 src/test/subscription/t/032_apply_delay.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index c1e4048054..bf3c05241c 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7873,6 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subminapplydelay</structfield> <type>int8</type>
+ </para>
+ <para>
+ The length of time (ms) to delay the application of changes.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subname</structfield> <type>name</type>
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 77574e2d4e..89cdc0a75b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4753,6 +4753,18 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
the <filename>postgresql.conf</filename> file or on the server
command line.
</para>
+ <para>
+ For time delayed logical replication, the apply worker sends a Standby
+ Status Update message to the corresponding publisher per the indicated
+ time of this parameter. Therefore, if this parameter is longer than
+ <literal>wal_sender_timeout</literal> on the publisher, then the
+ walsender doesn't get any update message during the delay and repeatedly
+ terminates due to the timeout errors. Hence, make sure this parameter is
+ shorter than the <literal>wal_sender_timeout</literal> of the publisher.
+ If this parameter is set to zero with time delayed replication, the
+ apply worker doesn't send any feedback messages during the
+ <literal>min_apply_delay</literal>.
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 54f48be87f..6407804547 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -247,6 +247,13 @@
target table.
</para>
+ <para>
+ The subscriber replication can be instructed to lag behind the publisher
+ side changes by specifying the <literal>min_apply_delay</literal>
+ subscription parameter. See <xref linkend="sql-createsubscription"/> for
+ details.
+ </para>
+
<sect2 id="logical-replication-subscription-slot">
<title>Replication Slot Management</title>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 1e8d72062b..d63aff1b90 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -213,8 +213,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
<literal>binary</literal>, <literal>streaming</literal>,
- <literal>disable_on_error</literal>, and
- <literal>origin</literal>.
+ <literal>disable_on_error</literal>,
+ <literal>origin</literal>, and
+ <literal>min_apply_delay</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index eba72c6af6..dc330e8db7 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -349,7 +349,47 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
- </variablelist></para>
+
+ <varlistentry>
+ <term><literal>min_apply_delay</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ By default, the subscriber applies changes as soon as possible. As
+ with the physical replication feature
+ (<xref linkend="guc-recovery-min-apply-delay"/>), it can be useful to
+ have a time-delayed logical replica. This parameter lets the user to
+ delay the application of changes by a specified amount of time. If this
+ value is specified without units, it is taken as milliseconds. The
+ default is zero(no delay).
+ </para>
+ <para>
+ The delay occurs only on WAL records for transaction begins and after
+ the initial table synchronization. It is possible that the
+ replication delay between publisher and subscriber exceeds the value
+ of this parameter, in which case no delay is added. Note that the
+ delay is calculated between the WAL time stamp as written on
+ publisher and the current time on the subscriber. Time spent in logical
+ decoding and in transferring the transaction may reduce the actual wait
+ time. If the system clocks on publisher and subscriber are not
+ synchronized, this may lead to apply changes earlier than expected,
+ but this is not a major issue because this parameter is typically much
+ larger than the time deviations between servers. Note that if this
+ parameter is set to a long delay, the replication will stop if the
+ replication slot falls behind the current LSN by more than
+ <link linkend="guc-max-slot-wal-keep-size"><literal>max_slot_wal_keep_size</literal></link>.
+ </para>
+ <warning>
+ <para>
+ Delaying the replication can mean there is a much longer time between making
+ a change on the publisher, and that change being committed on the subscriber.
+ This can have a big impact on synchronous replication.
+ See <xref linkend="guc-synchronous-commit"/>.
+ </para>
+ </warning>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
@@ -413,6 +453,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
published with different column lists are not supported.
</para>
+ <para>
+ Setting streaming to <literal>parallel</literal> mode and <literal>min_apply_delay</literal>
+ at the same time is not supported.
+ </para>
+
<para>
We allow non-existent publications to be specified so that users can add
those later. This means
@@ -472,6 +517,18 @@ CREATE SUBSCRIPTION mysub
PUBLICATION insert_only
WITH (enabled = false);
</programlisting></para>
+
+ <para>
+ Create a subscription to a remote server that replicates tables in
+ the <literal>mypub</literal> publication and starts replicating immediately
+ on commit. Pre-existing data is not copied. The application of changes is
+ delayed by 4 hours.
+<programlisting>
+CREATE SUBSCRIPTION mysub
+ CONNECTION 'host=192.0.2.4 port=5432 user=foo dbname=foodb'
+ PUBLICATION mypub
+ WITH (copy_data = false, min_apply_delay = '4h');
+</programlisting></para>
</refsect1>
<refsect1>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index a56ae311c3..c767cc1c3a 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -64,6 +64,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->oid = subid;
sub->dbid = subform->subdbid;
sub->skiplsn = subform->subskiplsn;
+ sub->minapplydelay = subform->subminapplydelay;
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 447c9b970f..4004fcd0c4 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1299,9 +1299,10 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
- subbinary, substream, subtwophasestate, subdisableonerr,
- subslotname, subsynccommit, subpublications, suborigin)
+GRANT SELECT (oid, subdbid, subskiplsn, subminapplydelay, subname, subowner,
+ subenabled, subbinary, substream, subtwophasestate,
+ subdisableonerr, subslotname, subsynccommit, subpublications,
+ suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index baff00dd74..b0ddd530e3 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -66,6 +66,7 @@
#define SUBOPT_DISABLE_ON_ERR 0x00000400
#define SUBOPT_LSN 0x00000800
#define SUBOPT_ORIGIN 0x00001000
+#define SUBOPT_MIN_APPLY_DELAY 0x00002000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -90,6 +91,7 @@ typedef struct SubOpts
bool disableonerr;
char *origin;
XLogRecPtr lsn;
+ int min_apply_delay;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -146,6 +148,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->disableonerr = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY))
+ opts->min_apply_delay = 0;
/* Parse options */
foreach(lc, stmt_options)
@@ -324,6 +328,43 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ strcmp(defel->defname, "min_apply_delay") == 0)
+ {
+ char *val,
+ *tmp;
+ Interval *interval;
+ int64 ms;
+
+ if (IsSet(opts->specified_opts, SUBOPT_MIN_APPLY_DELAY))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_MIN_APPLY_DELAY;
+ tmp = defGetString(defel);
+
+ /*
+ * If no unit was specified, then explicitly add 'ms' otherwise
+ * the interval_in function would assume 'seconds'.
+ */
+ if (strspn(tmp, "-0123456789 ") == strlen(tmp))
+ val = psprintf("%sms", tmp);
+ else
+ val = tmp;
+
+ interval = DatumGetIntervalP(DirectFunctionCall3(interval_in,
+ CStringGetDatum(val),
+ ObjectIdGetDatum(InvalidOid),
+ Int32GetDatum(-1)));
+
+ ms = interval2ms(interval);
+ if (ms < 0 || ms > INT_MAX)
+ ereport(ERROR,
+ errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg("%lld ms is outside the valid range for parameter \"%s\" (0 .. %d)",
+ (long long) ms, "min_apply_delay", INT_MAX));
+
+ opts->min_apply_delay = ms;
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -404,6 +445,17 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /* Test the combination of streaming mode and min_apply_delay */
+ if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ opts->min_apply_delay > 0)
+ {
+ if (opts->streaming == LOGICALREP_STREAM_PARALLEL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s and %s are mutually exclusive options",
+ "min_apply_delay > 0", "streaming = parallel"));
+ }
}
/*
@@ -560,7 +612,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
- SUBOPT_DISABLE_ON_ERR | SUBOPT_ORIGIN);
+ SUBOPT_DISABLE_ON_ERR | SUBOPT_ORIGIN |
+ SUBOPT_MIN_APPLY_DELAY);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -625,6 +678,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_oid - 1] = ObjectIdGetDatum(subid);
values[Anum_pg_subscription_subdbid - 1] = ObjectIdGetDatum(MyDatabaseId);
values[Anum_pg_subscription_subskiplsn - 1] = LSNGetDatum(InvalidXLogRecPtr);
+ values[Anum_pg_subscription_subminapplydelay - 1] = Int64GetDatum(opts.min_apply_delay);
values[Anum_pg_subscription_subname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
@@ -1054,7 +1108,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
- SUBOPT_ORIGIN);
+ SUBOPT_ORIGIN | SUBOPT_MIN_APPLY_DELAY);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1098,6 +1152,20 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
if (IsSet(opts.specified_opts, SUBOPT_STREAMING))
{
+ /*
+ * Test the combination of streaming mode and
+ * min_apply_delay
+ */
+ if (opts.streaming == LOGICALREP_STREAM_PARALLEL)
+ {
+ if ((IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) && opts.min_apply_delay > 0) ||
+ (!IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) && sub->minapplydelay > 0))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s mode for subscription with %s",
+ "streaming = parallel", "min_apply_delay"));
+ }
+
values[Anum_pg_subscription_substream - 1] =
CharGetDatum(opts.streaming);
replaces[Anum_pg_subscription_substream - 1] = true;
@@ -1111,6 +1179,26 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
= true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY))
+ {
+ /*
+ * Test the combination of streaming mode and
+ * min_apply_delay
+ */
+ if (opts.min_apply_delay > 0)
+ {
+ if ((IsSet(opts.specified_opts, SUBOPT_STREAMING) && opts.streaming == LOGICALREP_STREAM_PARALLEL) ||
+ (!IsSet(opts.specified_opts, SUBOPT_STREAMING) && sub->stream == LOGICALREP_STREAM_PARALLEL))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s for subscription in %s mode",
+ "min_apply_delay", "streaming = parallel"));
+ }
+ values[Anum_pg_subscription_subminapplydelay - 1] =
+ Int64GetDatum(opts.min_apply_delay);
+ replaces[Anum_pg_subscription_subminapplydelay - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index 2e5914d5d9..72e6f5ce84 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -704,7 +704,8 @@ pa_process_spooled_messages_if_required(void)
{
apply_spooled_messages(&MyParallelShared->fileset,
MyParallelShared->xid,
- InvalidXLogRecPtr);
+ InvalidXLogRecPtr,
+ 0);
pa_set_fileset_state(MyParallelShared, FS_EMPTY);
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 79cda39445..62f72027ac 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -316,6 +316,17 @@ static List *on_commit_wakeup_workers_subids = NIL;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/*
+ * In order to avoid walsender's timeout during time delayed replication,
+ * it's necessary to keep sending feedback messages during the delay from the
+ * worker process. Meanwhile, the feature delays the apply before starting the
+ * transaction and thus we don't write WALs for the suspended changes during
+ * the wait. Hence, in the case the worker process sends a feedback during the
+ * delay, avoid having positions of the flushed and apply LSN overwritten by
+ * the latest LSN.
+ */
+static XLogRecPtr last_received = InvalidXLogRecPtr;
+
/* fields valid only when processing streamed transaction */
static bool in_streamed_transaction = false;
@@ -386,10 +397,13 @@ static void stream_write_change(char action, StringInfo s);
static void stream_open_and_write_change(TransactionId xid, char action, StringInfo s);
static void stream_close_file(void);
-static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
+static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply,
+ bool in_delaying_apply);
static void DisableSubscriptionAndExit(void);
+static void maybe_delay_apply(TimestampTz finish_ts);
+
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
@@ -996,6 +1010,99 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
ExecStoreVirtualTuple(slot);
}
+/*
+ * When min_apply_delay parameter is set on the subscriber, we wait long enough
+ * to make sure a transaction is applied at least that interval behind the
+ * publisher.
+ *
+ * While the physical replication applies the delay at commit time, this
+ * feature applies the delay for the next transaction but before starting the
+ * transaction. This is mainly because keeping a transaction that conducted
+ * write operations open for a long time results in some issues such as bloat
+ * and locks.
+ *
+ * The min_apply_delay parameter will take effect only after all tables are in
+ * READY state.
+ *
+ * finish_ts is the commit/prepare time of both regular (non-streamed) and
+ * streamed transactions for time delayed replication.
+ */
+static void
+maybe_delay_apply(TimestampTz finish_ts)
+{
+ Assert(finish_ts > 0);
+
+ /* Nothing to do if no delay set */
+ if (MySubscription->minapplydelay <= 0)
+ return;
+
+ /*
+ * The min_apply_delay parameter is ignored until all tablesync workers
+ * have reached READY state. If we allow the delay during the catchup
+ * phase, once we reach the limit of tablesync workers, it will impose a
+ * delay for each subsequent worker. It means it will take a long time to
+ * finish the initial table synchronization.
+ */
+ if (!AllTablesyncsReady())
+ return;
+
+ while (true)
+ {
+ long diffms;
+
+ ResetLatch(MyLatch);
+
+ CHECK_FOR_INTERRUPTS();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /*
+ * Before calculating the time duration, reload the catalog if needed.
+ */
+ if (!in_remote_transaction && !in_streamed_transaction)
+ {
+ AcceptInvalidationMessages();
+ maybe_reread_subscription();
+ }
+
+ diffms = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ TimestampTzPlusMilliseconds(finish_ts, MySubscription->minapplydelay));
+
+ /*
+ * Exit without arming the latch if it's already past time to apply
+ * this transaction.
+ */
+ if (diffms <= 0)
+ break;
+
+ elog(DEBUG2, "logical replication apply delay: %ld ms", diffms);
+
+ /*
+ * Call send_feedback() to prevent the publisher from exiting by
+ * timeout during the delay, when wal_receiver_status_interval is
+ * available.
+ */
+ if (wal_receiver_status_interval > 0
+ && diffms > wal_receiver_status_interval * 1000)
+ {
+ WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ (long) wal_receiver_status_interval * 1000,
+ WAIT_EVENT_RECOVERY_APPLY_DELAY);
+ send_feedback(last_received, true, false, true);
+ }
+ else
+ WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ diffms,
+ WAIT_EVENT_RECOVERY_APPLY_DELAY);
+ }
+}
+
/*
* Handle BEGIN message.
*/
@@ -1007,6 +1114,9 @@ apply_handle_begin(StringInfo s)
logicalrep_read_begin(s, &begin_data);
set_apply_error_context_xact(begin_data.xid, begin_data.final_lsn);
+ /* Should we delay the current transaction? */
+ maybe_delay_apply(begin_data.committime);
+
remote_final_lsn = begin_data.final_lsn;
maybe_start_skipping_changes(begin_data.final_lsn);
@@ -1061,6 +1171,9 @@ apply_handle_begin_prepare(StringInfo s)
logicalrep_read_begin_prepare(s, &begin_data);
set_apply_error_context_xact(begin_data.xid, begin_data.prepare_lsn);
+ /* Should we delay the current prepared transaction? */
+ maybe_delay_apply(begin_data.prepare_time);
+
remote_final_lsn = begin_data.prepare_lsn;
maybe_start_skipping_changes(begin_data.prepare_lsn);
@@ -1308,7 +1421,8 @@ apply_handle_stream_prepare(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset,
- prepare_data.xid, prepare_data.prepare_lsn);
+ prepare_data.xid, prepare_data.prepare_lsn,
+ prepare_data.prepare_time);
/* Mark the transaction as prepared. */
apply_handle_prepare_internal(&prepare_data);
@@ -1998,10 +2112,13 @@ ensure_last_message(FileSet *stream_fileset, TransactionId xid, int fileno,
/*
* Common spoolfile processing.
+ *
+ * The commit/prepare time for streaming transaction is required to achieve time
+ * delayed replication.
*/
void
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn, TimestampTz finish_ts)
{
StringInfoData s2;
int nchanges;
@@ -2012,6 +2129,21 @@ apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
int fileno;
off_t offset;
+ /*
+ * Should we delay the current transaction?
+ *
+ * Unlike the regular (non-streamed) cases, the delay is applied in a
+ * STREAM COMMIT/STREAM PREPARE message for streamed transactions. The
+ * STREAM START message does not contain a commit/prepare time (it will be
+ * available when the in-progress transaction finishes). Hence, it's not
+ * appropriate to apply a delay at that time.
+ *
+ * It's not allowed to execute time delayed replication with parallel
+ * apply feature.
+ */
+ if (!am_parallel_apply_worker())
+ maybe_delay_apply(finish_ts);
+
if (!am_parallel_apply_worker())
maybe_start_skipping_changes(lsn);
@@ -2171,7 +2303,7 @@ apply_handle_stream_commit(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset, xid,
- commit_data.commit_lsn);
+ commit_data.commit_lsn, commit_data.committime);
apply_handle_commit_internal(&commit_data);
@@ -3444,7 +3576,7 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
* Apply main loop.
*/
static void
-LogicalRepApplyLoop(XLogRecPtr last_received)
+LogicalRepApplyLoop(void)
{
TimestampTz last_recv_timestamp = GetCurrentTimestamp();
bool ping_sent = false;
@@ -3565,7 +3697,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
if (last_received < end_lsn)
last_received = end_lsn;
- send_feedback(last_received, reply_requested, false);
+ send_feedback(last_received, reply_requested, false, false);
UpdateWorkerStats(last_received, timestamp, true);
}
/* other message types are purposefully ignored */
@@ -3578,7 +3710,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/* confirm all writes so far */
- send_feedback(last_received, false, false);
+ send_feedback(last_received, false, false, false);
if (!in_remote_transaction && !in_streamed_transaction)
{
@@ -3675,7 +3807,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
}
- send_feedback(last_received, requestReply, requestReply);
+ send_feedback(last_received, requestReply, requestReply, false);
/*
* Force reporting to ensure long idle periods don't lead to
@@ -3705,7 +3837,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
* to send a response to avoid timeouts.
*/
static void
-send_feedback(XLogRecPtr recvpos, bool force, bool requestReply)
+send_feedback(XLogRecPtr recvpos, bool force, bool requestReply, bool in_delaying_apply)
{
static StringInfo reply_message = NULL;
static TimestampTz send_time = 0;
@@ -3735,8 +3867,15 @@ send_feedback(XLogRecPtr recvpos, bool force, bool requestReply)
/*
* No outstanding transactions to flush, we can report the latest received
* position. This is important for synchronous replication.
+ *
+ * During the delay of time delayed replication, do not tell the publisher
+ * that the received latest LSN is already applied and flushed at this
+ * stage, since we don't apply the transaction yet. If we do so, it leads
+ * to a wrong assumption of logical replication progress on the publisher
+ * side. Here, we just send a feedback message to avoid publisher's
+ * timeout during the delay.
*/
- if (!have_pending_txes)
+ if (!have_pending_txes && !in_delaying_apply)
flushpos = writepos = recvpos;
if (writepos < last_writepos)
@@ -4362,11 +4501,11 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname)
* of system resource error and are not repeatable.
*/
static void
-start_apply(XLogRecPtr origin_startpos)
+start_apply(void)
{
PG_TRY();
{
- LogicalRepApplyLoop(origin_startpos);
+ LogicalRepApplyLoop();
}
PG_CATCH();
{
@@ -4653,7 +4792,8 @@ ApplyWorkerMain(Datum main_arg)
}
/* Run the main loop. */
- start_apply(origin_startpos);
+ last_received = origin_startpos;
+ start_apply();
proc_exit(0);
}
diff --git a/src/backend/utils/adt/timestamp.c b/src/backend/utils/adt/timestamp.c
index 928c330897..422e6ad0fa 100644
--- a/src/backend/utils/adt/timestamp.c
+++ b/src/backend/utils/adt/timestamp.c
@@ -2431,6 +2431,35 @@ interval_cmp_internal(const Interval *interval1, const Interval *interval2)
return int128_compare(span1, span2);
}
+/*
+ * Returns the number of milliseconds in the specified Interval.
+ */
+int64
+interval2ms(const Interval *interval)
+{
+ int64 days;
+ int64 ms;
+ int64 result;
+
+ days = interval->month * INT64CONST(30);
+ days += interval->day;
+
+ /* Detect whether the value of interval can cause an overflow */
+ if (pg_mul_s64_overflow(days, MSECS_PER_DAY, &result))
+ ereport(ERROR,
+ errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg("bigint out of range"));
+
+ /* Adds portion time (in ms) to the previous result */
+ ms = interval->time / INT64CONST(1000);
+ if (pg_add_s64_overflow(result, ms, &result))
+ ereport(ERROR,
+ errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg("bigint out of range"));
+
+ return result;
+}
+
Datum
interval_eq(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 2c0a969972..16d3b003fe 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4494,6 +4494,7 @@ getSubscriptions(Archive *fout)
int i_subsynccommit;
int i_subpublications;
int i_subbinary;
+ int i_subminapplydelay;
int i,
ntups;
@@ -4546,9 +4547,14 @@ getSubscriptions(Archive *fout)
LOGICALREP_TWOPHASE_STATE_DISABLED);
if (fout->remoteVersion >= 160000)
- appendPQExpBufferStr(query, " s.suborigin\n");
+ appendPQExpBufferStr(query,
+ " s.suborigin,\n"
+ " s.subminapplydelay\n");
else
- appendPQExpBuffer(query, " '%s' AS suborigin\n", LOGICALREP_ORIGIN_ANY);
+ {
+ appendPQExpBuffer(query, " '%s' AS suborigin,\n", LOGICALREP_ORIGIN_ANY);
+ appendPQExpBufferStr(query, " 0 AS subminapplydelay\n");
+ }
appendPQExpBufferStr(query,
"FROM pg_subscription s\n"
@@ -4576,6 +4582,7 @@ getSubscriptions(Archive *fout)
i_subtwophasestate = PQfnumber(res, "subtwophasestate");
i_subdisableonerr = PQfnumber(res, "subdisableonerr");
i_suborigin = PQfnumber(res, "suborigin");
+ i_subminapplydelay = PQfnumber(res, "subminapplydelay");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4606,6 +4613,8 @@ getSubscriptions(Archive *fout)
subinfo[i].subdisableonerr =
pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
subinfo[i].suborigin = pg_strdup(PQgetvalue(res, i, i_suborigin));
+ subinfo[i].subminapplydelay =
+ strtoi64(PQgetvalue(res, i, i_subminapplydelay), NULL, 10);
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -4687,6 +4696,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
+ if (subinfo->subminapplydelay > 0)
+ appendPQExpBuffer(query, ", min_apply_delay = '" INT64_FORMAT " ms'", subinfo->subminapplydelay);
+
appendPQExpBufferStr(query, ");\n");
if (subinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index e7cbd8d7ed..e2525f70ab 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -661,6 +661,7 @@ typedef struct _SubscriptionInfo
char *subdisableonerr;
char *suborigin;
char *subsynccommit;
+ int64 subminapplydelay;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index c8a0bb7b3a..8a27063bed 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6472,7 +6472,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false, false, false, false};
+ false, false, false, false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6527,10 +6527,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Two-phase commit"),
gettext_noop("Disable on error"));
+ /* Origin and min_apply_delay are only supported in v16 and higher */
if (pset.sversion >= 160000)
appendPQExpBuffer(&buf,
- ", suborigin AS \"%s\"\n",
- gettext_noop("Origin"));
+ ", suborigin AS \"%s\"\n"
+ ", subminapplydelay AS \"%s\"\n",
+ gettext_noop("Origin"),
+ gettext_noop("Min apply delay (ms)"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 23750ea5fb..19d2f90dc0 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1925,7 +1925,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "origin", "slot_name",
+ COMPLETE_WITH("binary", "disable_on_error", "min_apply_delay", "origin", "slot_name",
"streaming", "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
@@ -3247,7 +3247,7 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "origin", "slot_name",
+ "disable_on_error", "enabled", "min_apply_delay", "origin", "slot_name",
"streaming", "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index b0f2a1705d..078495cbf0 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -70,6 +70,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
XLogRecPtr subskiplsn; /* All changes finished at this LSN are
* skipped */
+ int64 subminapplydelay; /* Replication apply delay */
+
NameData subname; /* Name of the subscription */
Oid subowner BKI_LOOKUP(pg_authid); /* Owner of the subscription */
@@ -120,6 +122,7 @@ typedef struct Subscription
* in */
XLogRecPtr skiplsn; /* All changes finished at this LSN are
* skipped */
+ int64 minapplydelay; /* Replication apply delay */
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
diff --git a/src/include/datatype/timestamp.h b/src/include/datatype/timestamp.h
index 21a37e21e9..8b368af299 100644
--- a/src/include/datatype/timestamp.h
+++ b/src/include/datatype/timestamp.h
@@ -127,6 +127,8 @@ struct pg_itm_in
#define SECS_PER_MINUTE 60
#define MINS_PER_HOUR 60
+#define MSECS_PER_DAY INT64CONST(86400000)
+
#define USECS_PER_DAY INT64CONST(86400000000)
#define USECS_PER_HOUR INT64CONST(3600000000)
#define USECS_PER_MINUTE INT64CONST(60000000)
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index db891eea8a..4ef132d243 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -255,7 +255,7 @@ extern void stream_stop_internal(TransactionId xid);
/* Common streaming function to apply all the spooled messages */
extern void apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn);
+ XLogRecPtr lsn, TimestampTz finish_ts);
extern void apply_dispatch(StringInfo s);
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index 42f802bb9d..534051fe13 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -102,6 +102,8 @@ extern bool TimestampDifferenceExceeds(TimestampTz start_time,
TimestampTz stop_time,
int msec);
+extern int64 interval2ms(const Interval *interval);
+
extern TimestampTz time_t_to_timestamptz(pg_time_t tm);
extern pg_time_t timestamptz_to_time_t(TimestampTz t);
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 1ed6f4c39c..7b16c80926 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -114,18 +114,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -135,10 +135,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -155,10 +155,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -167,10 +167,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -202,10 +202,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -239,19 +239,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -263,27 +263,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -298,10 +298,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more then once
@@ -316,10 +316,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -355,10 +355,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -367,10 +367,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -380,10 +380,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,20 +396,61 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail -- min_apply_delay must be a non-negative integer
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = foo);
+ERROR: invalid input syntax for type interval: "foo"
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = -1);
+ERROR: -1 ms is outside the valid range for parameter "min_apply_delay" (0 .. 2147483647)
+-- fail - utilizing streaming = parallel with time delayed replication is not supported.
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = parallel, min_apply_delay = '1 day');
+ERROR: min_apply_delay > 0 and streaming = parallel are mutually exclusive options
+-- success -- value without unit is taken as milliseconds
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = 123);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 123 | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+-- success -- interval is converted into ms and stored as integer
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = '4h 27min 35s');
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 16055000 | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+-- fail - alter subscription with streaming = parallel should fail when time delayed replication is set.
+ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
+ERROR: cannot enable streaming = parallel mode for subscription with min_apply_delay
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - alter subscription with min_apply_delay should fail when streaming = parallel is set.
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = parallel);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 123);
+ERROR: cannot enable min_apply_delay for subscription in streaming = parallel mode
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
RESET SESSION AUTHORIZATION;
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7991abfe8f..ba822d4c29 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -279,6 +279,31 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail -- min_apply_delay must be a non-negative integer
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = foo);
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = -1);
+-- fail - utilizing streaming = parallel with time delayed replication is not supported.
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = parallel, min_apply_delay = '1 day');
+
+-- success -- value without unit is taken as milliseconds
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = 123);
+\dRs+
+
+-- success -- interval is converted into ms and stored as integer
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = '4h 27min 35s');
+\dRs+
+
+-- fail - alter subscription with streaming = parallel should fail when time delayed replication is set.
+ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+-- fail - alter subscription with min_apply_delay should fail when streaming = parallel is set.
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = parallel);
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 123);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index 3db0fdfd96..a186876eb4 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -38,6 +38,7 @@ tests += {
't/029_on_error.pl',
't/030_origin.pl',
't/031_column_list.pl',
+ 't/032_apply_delay.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/032_apply_delay.pl b/src/test/subscription/t/032_apply_delay.pl
new file mode 100644
index 0000000000..b21de1ff63
--- /dev/null
+++ b/src/test/subscription/t/032_apply_delay.pl
@@ -0,0 +1,170 @@
+
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test replication apply delay
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Create publisher node.
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node.
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = debug2");
+$node_subscriber->start;
+
+# Create some preexisting content on publisher.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar, c timestamptz DEFAULT now())");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber.
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)"
+);
+
+# Setup logical replication.
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+# column c must not be published because we want to compare the time difference.
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE test_tab (a, b)");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on, min_apply_delay = '3s')"
+);
+
+# Wait for initial table sync to finish.
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Check log starting now for logical replication apply delay.
+my $log_location = -s $node_subscriber->logfile;
+
+my $result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM test_tab");
+is($result, qq(2|1|2), 'check initial data was copied to subscriber');
+
+# New row to trigger apply delay.
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (3, 'baz')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (4, 'abc')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (5, 'def')");
+
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM test_tab");
+is($result, qq(5|1|5), 'check if the new rows were applied to subscriber');
+
+check_apply_delay_log("logical replication apply delay", "2000");
+
+check_apply_delay_time('5', '3');
+
+# Test streamed transaction.
+# Insert, update and delete enough rows to exceed 64kB limit.
+$node_publisher->safe_psql(
+ 'postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a, 2) = 0;
+DELETE FROM test_tab WHERE mod(a, 3) = 0;
+COMMIT;
+});
+
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM test_tab");
+is($result, qq(3334|1|5000), 'check if the new rows were applied to subscriber');
+
+check_apply_delay_log("logical replication apply delay", "2000");
+
+check_apply_delay_time('5000', '3');
+
+# Test ALTER SUBSCRIPTION. Delay 86460 seconds (1 day 1 minute).
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (min_apply_delay = 86460000)"
+);
+
+# New row to trigger apply delay.
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (0, 'foobar')");
+
+check_apply_delay_log("logical replication apply delay", "80000000");
+
+# Disable subscription. worker should die immediately.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub DISABLE;"
+);
+
+# Wait until worker dies.
+my $sub_query =
+ "SELECT count(1) = 0 FROM pg_stat_subscription WHERE subname = 'tap_sub' AND pid IS NOT NULL;";
+$node_subscriber->poll_query_until('postgres', $sub_query)
+ or die "Timed out while waiting for subscriber to die";
+
+$node_subscriber->stop;
+$node_publisher->stop;
+
+done_testing();
+
+sub check_apply_delay_log
+{
+ my ($message, $expected) = @_;
+ $expected = 0 unless defined $expected;
+
+ my $old_log_location = $log_location;
+
+ $log_location = $node_subscriber->wait_for_log(qr/$message/, $log_location);
+
+ cmp_ok($log_location, '>', $old_log_location,
+ "logfile contains triggered logical replication apply delay"
+ );
+
+ if ($expected > 0)
+ {
+ # Get the delay time in the server log.
+ my $contents = slurp_file($node_subscriber->logfile, $old_log_location);
+ $contents =~
+ qr/logical replication apply delay: (\d+) ms/
+ or die "could not get time";
+ my $logged_delay = $1;
+
+ # Is it larger than expected ?
+ cmp_ok($logged_delay, '>', $expected,
+ "The wait time of the apply worker is long enough expectedly"
+ );
+ }
+}
+
+sub check_apply_delay_time
+{
+ my ($primary_key, $expected_diffs) = @_;
+
+ my $inserted_time_on_pub = $node_publisher->safe_psql('postgres', qq[
+ SELECT extract(epoch from c) FROM test_tab WHERE a = $primary_key;
+ ]);
+
+ my $inserted_time_on_sub = $node_subscriber->safe_psql('postgres', qq[
+ SELECT extract(epoch from c) FROM test_tab WHERE a = $primary_key;
+ ]);
+
+ cmp_ok($inserted_time_on_sub - $inserted_time_on_pub, '>', $expected_diffs,
+ "The tuple on the subscriber was modified later than the publisher");
+}
--
2.30.0
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-14 06:27 vignesh C <[email protected]>
parent: Takamichi Osumi (Fujitsu) <[email protected]>
0 siblings, 1 reply; 40+ messages in thread
From: vignesh C @ 2023-01-14 06:27 UTC (permalink / raw)
To: Takamichi Osumi (Fujitsu) <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>
On Thu, 12 Jan 2023 at 21:09, Takamichi Osumi (Fujitsu)
<[email protected]> wrote:
>
> On Thursday, January 12, 2023 12:04 PM Kyotaro Horiguchi <[email protected]> wrote:
> > At Wed, 11 Jan 2023 12:46:24 +0000, "Hayato Kuroda (Fujitsu)"
> > <[email protected]> wrote in
> > > them. Which version is better?
> >
> >
> > Some comments by a quick loock, different from the above.
> Horiguchi-san, thanks for your review !
>
>
> > + CONNECTION 'host=192.168.1.50 port=5432 user=foo
> > dbname=foodb'
> >
> > I understand that we (not PG people, but IT people) are supposed to use in
> > documents a certain set of special addresses that is guaranteed not to be
> > routed in the field.
> >
> > > TEST-NET-1 : 192.0.2.0/24
> > > TEST-NET-2 : 198.51.100.0/24
> > > TEST-NET-3 : 203.0.113.0/24
> >
> > (I found 192.83.123.89 in the postgres_fdw doc, but it'd be another issue..)
> Fixed. If necessary we can create another thread for this.
>
> > + if (strspn(tmp, "-0123456789 ") == strlen(tmp))
> >
> > Do we need to bother spending another memory block for apparent non-digits
> > here?
> Yes. The characters are necessary to handle an issue reported in [1].
> The issue happened if the user inputs a negative value,
> then the length comparison became different between strspn and strlen
> and the input value was recognized as seconds, when
> the unit wasn't described. This led to a wrong error message for the user.
>
> Those addition of such characters solve the issue.
>
> > + errmsg(INT64_FORMAT " ms
> > is outside the valid range for parameter
> > +\"%s\"",
> >
> > We don't use INT64_FORMAT in translatable message strings. Cast then
> > use %lld instead.
> Thanks for teaching us. Fixed.
>
> > This message looks unfriendly as it doesn't suggest the valid range, and it
> > shows the input value in a different unit from what was in the input. A I think we
> > can spell it as "\"%s\" is outside the valid range for subsciription parameter
> > \"%s\" (0 .. <INT_MAX> in millisecond)"
> Makes sense. I incorporated the valid range with the aligned format of recovery_min_apply_delay.
> FYI, the physical replication's GUC doesn't write the unites for the range like below.
> I followed and applied this style.
>
> ---
> LOG: -1 ms is outside the valid range for parameter "recovery_min_apply_delay" (0 .. 2147483647)
> FATAL: configuration file "/home/k5user/new/pg/l/make_v15/slave/postgresql.conf" contains errors
> ---
>
> > + int64 min_apply_delay;
> > ..
> > + if (ms < 0 || ms > INT_MAX)
> >
> > Why is the variable wider than required?
> You are right. Fixed.
>
> > + errmsg("%s and %s are mutually
> > exclusive options",
> > + "min_apply_delay > 0",
> > "streaming = parallel"));
> >
> > Mmm. Couldn't we refuse 0 as min_apply_delay?
> Sorry, the previous patch's behavior wasn't consistent with this error message.
>
> In the previous patch, if we conducted alter subscription
> with stream = parallel and min_apply_delay = 0 (from a positive value) at the same time,
> the alter command failed, although this should succeed by this time-delayed feature specification.
> We fixed this part accordingly by some more tests in AlterSubscription().
>
> By the way, we should allow users to change min_apply_dealy to 0
> whenever they want from different value. Then, we didn't restrict
> this kind of operation.
>
> > + sub->minapplydelay > 0)
> > ...
> > + if (opts.min_apply_delay > 0 &&
> >
> > Is there any reason for the differenciation?
> Yes. The former is the object for an existing subscription configuration.
> For example, if we alter subscription with setting streaming = 'parallel'
> for a subscription created with min_apply_delay = '1 day', we
> need to reject the alter command. The latter is new settings.
>
>
> > +
> > errmsg("cannot set %s for subscription with %s",
> > +
> > "streaming = parallel", "min_apply_delay > 0"));
> >
> > I think that this shoud be more like human-speking. Say, "cannot enable
> > min_apply_delay for subscription in parallel streaming mode" or something..
> > The same is applicable to the nearby message.
> Reworded the error messages. Please check.
>
> > +static void maybe_delay_apply(TimestampTz ts);
> >
> > apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
> > - XLogRecPtr lsn)
> > + XLogRecPtr lsn, TimestampTz ts)
> >
> > "ts" looks too generic. Couldn't it be more specific?
> > We need a explanation for the parameter in the function comment.
> Changed it to finish_ts, since it indicates commit/prepare time.
> This terminology should be aligned with finish lsn.
>
> > + if (!am_parallel_apply_worker())
> > + {
> > + Assert(ts > 0);
> > + maybe_delay_apply(ts);
> >
> > It seems to me better that the if condition and assertion are checked inside
> > maybe_delay_apply().
> Fixed.
>
Thanks for the updated patch, Few comments:
1) Since the min_apply_delay = 3, but you have specified 2s, there
might be a possibility that it can log delay as 1000ms due to
pub/sub/network delay and the test can fail randomly, If we cannot
ensure this log file value, check_apply_delay_time verification alone
should be sufficient.
+is($result, qq(5|1|5), 'check if the new rows were applied to subscriber');
+
+check_apply_delay_log("logical replication apply delay", "2000");
2) I'm not sure if this will add any extra coverage as the altering
value of min_apply_delay is already tested in the regression, if so
this test can be removed:
+# Test ALTER SUBSCRIPTION. Delay 86460 seconds (1 day 1 minute).
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (min_apply_delay = 86460000)"
+);
+
+# New row to trigger apply delay.
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (0, 'foobar')");
+
+check_apply_delay_log("logical replication apply delay", "80000000");
3) We generally keep the subroutines before the tests, it can be kept
accordingly:
3.a)
+sub check_apply_delay_log
+{
+ my ($message, $expected) = @_;
+ $expected = 0 unless defined $expected;
+
+ my $old_log_location = $log_location;
3.b)
+sub check_apply_delay_time
+{
+ my ($primary_key, $expected_diffs) = @_;
+
+ my $inserted_time_on_pub = $node_publisher->safe_psql('postgres', qq[
+ SELECT extract(epoch from c) FROM test_tab WHERE a =
$primary_key;
+ ]);
+
4) typo "more then once" should be "more than once"
+ regress_testsub | regress_subscription_user | f |
{testpub,testpub1,testpub2} | f | off | d |
f | any | 0 | off
| dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more then once
@@ -316,10 +316,10 @@ ERROR: publication "testpub3" is not in
subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1,
testpub2 WITH (refresh = false);
\dRs+
5) This can be changed to "Is it larger than expected?"
+ # Is it larger than expected ?
+ cmp_ok($logged_delay, '>', $expected,
+ "The wait time of the apply worker is long
enough expectedly"
+ );
6) 2022 should be changed to 2023
+++ b/src/test/subscription/t/032_apply_delay.pl
@@ -0,0 +1,170 @@
+
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test replication apply delay
7) Termination full stop is not required for single line comments:
7.a)
+use Test::More;
+
+# Create publisher node.
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
7.b) +
+# Create subscriber node.
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
7.c) +
+# Create some preexisting content on publisher.
+$node_publisher->safe_psql('postgres',
7.d) similarly in rest of the files
8) Is it possible to add one test for spooling also?
Regards,
Vignesh
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-17 11:00 Takamichi Osumi (Fujitsu) <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 2 replies; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-17 11:00 UTC (permalink / raw)
To: 'vignesh C' <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>
Hi,
On Saturday, January 14, 2023 3:27 PM vignesh C <[email protected]> wrote:
> 1) Since the min_apply_delay = 3, but you have specified 2s, there might be a
> possibility that it can log delay as 1000ms due to pub/sub/network delay and
> the test can fail randomly, If we cannot ensure this log file value,
> check_apply_delay_time verification alone should be sufficient.
> +is($result, qq(5|1|5), 'check if the new rows were applied to
> +subscriber');
> +
> +check_apply_delay_log("logical replication apply delay", "2000");
You are right. Removed the left-time check of the 1st call of check_apply_delay_log().
> 2) I'm not sure if this will add any extra coverage as the altering value of
> min_apply_delay is already tested in the regression, if so this test can be
> removed:
> +# Test ALTER SUBSCRIPTION. Delay 86460 seconds (1 day 1 minute).
> +$node_subscriber->safe_psql('postgres',
> + "ALTER SUBSCRIPTION tap_sub SET (min_apply_delay =
> 86460000)"
> +);
> +
> +# New row to trigger apply delay.
> +$node_publisher->safe_psql('postgres',
> + "INSERT INTO test_tab VALUES (0, 'foobar')");
> +
> +check_apply_delay_log("logical replication apply delay", "80000000");
While addressing this point, I've noticed that there is a
behavior difference between physical replication's recovery_min_apply_delay
and this feature when stopping the replication during delays.
At present, in the latter case,
the apply worker exits without applying the suspended transaction
after ALTER SUBSCRIPTION DISABLE command for the subscription.
Meanwhile, there is no "disabling" command for physical replication,
but I checked the behavior about what happens for promoting a secondary
during the delay of recovery_min_apply_delay for physical replication as one example.
The transaction has become visible even in the promoting in the middle of delay.
I'm not sure if I should make the time-delayed LR aligned with this behavior.
Does someone has an opinion for this ?
By the way, the above test code can be used for the test case
when the apply worker is in a delay but the transaction has been canceled by
ALTER SUBSCRIPTION DISABLE command. So, I didn't remove it at this stage.
> 3) We generally keep the subroutines before the tests, it can be kept
> accordingly:
> 3.a)
> +sub check_apply_delay_log
> +{
> + my ($message, $expected) = @_;
> + $expected = 0 unless defined $expected;
> +
> + my $old_log_location = $log_location;
>
> 3.b)
> +sub check_apply_delay_time
> +{
> + my ($primary_key, $expected_diffs) = @_;
> +
> + my $inserted_time_on_pub = $node_publisher->safe_psql('postgres',
> qq[
> + SELECT extract(epoch from c) FROM test_tab WHERE a =
> $primary_key;
> + ]);
> +
Fixed.
> 4) typo "more then once" should be "more than once"
> + regress_testsub | regress_subscription_user | f |
> {testpub,testpub1,testpub2} | f | off | d |
> f | any | 0 | off
> | dbname=regress_doesnotexist | 0/0
> (1 row)
>
> -- fail - publication used more then once @@ -316,10 +316,10 @@ ERROR:
> publication "testpub3" is not in subscription "regress_testsub"
> -- ok - delete publications
> ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1,
> testpub2 WITH (refresh = false);
> \dRs+
This was an existing typo on HEAD. Addressed in other thread in [1].
> 5) This can be changed to "Is it larger than expected?"
> + # Is it larger than expected ?
> + cmp_ok($logged_delay, '>', $expected,
> + "The wait time of the apply worker is long
> enough expectedly"
> + );
Fixed.
> 6) 2022 should be changed to 2023
> +++ b/src/test/subscription/t/032_apply_delay.pl
> @@ -0,0 +1,170 @@
> +
> +# Copyright (c) 2022, PostgreSQL Global Development Group
> +
> +# Test replication apply delay
Fixed.
> 7) Termination full stop is not required for single line comments:
> 7.a)
> +use Test::More;
> +
> +# Create publisher node.
> +my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
>
> 7.b) +
> +# Create subscriber node.
> +my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
>
> 7.c) +
> +# Create some preexisting content on publisher.
> +$node_publisher->safe_psql('postgres',
>
> 7.d) similarly in rest of the files
Removed the periods for single line comments.
> 8) Is it possible to add one test for spooling also?
There is a streaming transaction case in the TAP test already.
I conducted some minor comment modifications along with above changes.
Kindly have a look at the v16.
[1] - https://www.postgresql.org/message-id/flat/TYCPR01MB83737EA140C79B7D099F65E8EDC69%40TYCPR01MB8373.jp...
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v16-0001-Time-delayed-logical-replication-subscriber.patch (80.5K, ../../TYCPR01MB8373F3E780D5BB60E441ECF3EDC69@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v16-0001-Time-delayed-logical-replication-subscriber.patch)
download | inline diff:
From 47af0bc714caccb39005386b495fc2c9460d25de Mon Sep 17 00:00:00 2001
From: Takamichi Osumi <[email protected]>
Date: Tue, 17 Jan 2023 10:03:30 +0000
Subject: [PATCH v16] Time-delayed logical replication subscriber
Similar to physical replication, a time-delayed copy of the data for
logical replication is useful for some scenarios (particularly to fix
errors that might cause data loss).
If the subscription sets min_apply_delay parameter, the logical
replication worker will delay the transaction commit for min_apply_delay
milliseconds.
The delay is calculated between the WAL time stamp and the current time
on the subscriber.
The delay occurs only on WAL records for transaction begins. The main
reason is to avoid keeping a transaction open for a long time. Regular
and prepared transactions are covered. Streamed transactions are also
covered.
Prohibit the combination of this feature and parallel streaming mode.
Author: Euler Taveira
Discussion: https://postgr.es/m/CAB-JLwYOYwL=XTyAXKiH5CtM_Vm8KjKh7aaitCKvmCh4rzr5pQ@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/config.sgml | 12 ++
doc/src/sgml/logical-replication.sgml | 7 +
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 59 +++++-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 7 +-
src/backend/commands/subscriptioncmds.c | 89 ++++++++-
.../replication/logical/applyparallelworker.c | 3 +-
src/backend/replication/logical/worker.c | 166 ++++++++++++++--
src/backend/utils/adt/timestamp.c | 29 +++
src/bin/pg_dump/pg_dump.c | 16 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 9 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 3 +
src/include/datatype/timestamp.h | 2 +
src/include/replication/worker_internal.h | 2 +-
src/include/utils/timestamp.h | 2 +
src/test/regress/expected/subscription.out | 185 +++++++++++-------
src/test/regress/sql/subscription.sql | 25 +++
src/test/subscription/meson.build | 1 +
src/test/subscription/t/032_apply_delay.pl | 182 +++++++++++++++++
23 files changed, 717 insertions(+), 102 deletions(-)
create mode 100644 src/test/subscription/t/032_apply_delay.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index c1e4048054..bf3c05241c 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7873,6 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subminapplydelay</structfield> <type>int8</type>
+ </para>
+ <para>
+ The length of time (ms) to delay the application of changes.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subname</structfield> <type>name</type>
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 77574e2d4e..8258e251df 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4753,6 +4753,18 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
the <filename>postgresql.conf</filename> file or on the server
command line.
</para>
+ <para>
+ For time-delayed logical replication, the apply worker sends a Standby
+ Status Update message to the corresponding publisher per the indicated
+ time of this parameter. Therefore, if this parameter is longer than
+ <literal>wal_sender_timeout</literal> on the publisher, then the
+ walsender doesn't get any update message during the delay and repeatedly
+ terminates due to the timeout errors. Hence, make sure this parameter is
+ shorter than the <literal>wal_sender_timeout</literal> of the publisher.
+ If this parameter is set to zero with time-delayed replication, the
+ apply worker doesn't send any feedback messages during the
+ <literal>min_apply_delay</literal>.
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 54f48be87f..6407804547 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -247,6 +247,13 @@
target table.
</para>
+ <para>
+ The subscriber replication can be instructed to lag behind the publisher
+ side changes by specifying the <literal>min_apply_delay</literal>
+ subscription parameter. See <xref linkend="sql-createsubscription"/> for
+ details.
+ </para>
+
<sect2 id="logical-replication-subscription-slot">
<title>Replication Slot Management</title>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index ad93553a1d..1c6e9dd2d1 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -213,8 +213,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
<literal>binary</literal>, <literal>streaming</literal>,
- <literal>disable_on_error</literal>, and
- <literal>origin</literal>.
+ <literal>disable_on_error</literal>,
+ <literal>origin</literal>, and
+ <literal>min_apply_delay</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index eba72c6af6..ad97914dc8 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -349,7 +349,47 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
- </variablelist></para>
+
+ <varlistentry>
+ <term><literal>min_apply_delay</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ By default, the subscriber applies changes as soon as possible. As
+ with the physical replication feature
+ (<xref linkend="guc-recovery-min-apply-delay"/>), it can be useful to
+ have a time-delayed logical replica. This parameter lets the user to
+ delay the application of changes by a specified amount of time. If this
+ value is specified without units, it is taken as milliseconds. The
+ default is zero(no delay).
+ </para>
+ <para>
+ The delay occurs only on WAL records for transaction begins and after
+ the initial table synchronization. It is possible that the
+ replication delay between publisher and subscriber exceeds the value
+ of this parameter, in which case no delay is added. Note that the
+ delay is calculated between the WAL time stamp as written on
+ publisher and the current time on the subscriber. Time spent in logical
+ decoding and in transferring the transaction may reduce the actual wait
+ time. If the system clocks on publisher and subscriber are not
+ synchronized, this may lead to apply changes earlier than expected,
+ but this is not a major issue because this parameter is typically much
+ larger than the time deviations between servers. Note that if this
+ parameter is set to a long delay, the replication will stop if the
+ replication slot falls behind the current LSN by more than
+ <link linkend="guc-max-slot-wal-keep-size"><literal>max_slot_wal_keep_size</literal></link>.
+ </para>
+ <warning>
+ <para>
+ Delaying the replication can mean there is a much longer time between making
+ a change on the publisher, and that change being committed on the subscriber.
+ This can have a big impact on synchronous replication.
+ See <xref linkend="guc-synchronous-commit"/>.
+ </para>
+ </warning>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
@@ -413,6 +453,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
published with different column lists are not supported.
</para>
+ <para>
+ Setting streaming to <literal>parallel</literal> mode and <literal>min_apply_delay</literal>
+ simultaneously is not supported.
+ </para>
+
<para>
We allow non-existent publications to be specified so that users can add
those later. This means
@@ -472,6 +517,18 @@ CREATE SUBSCRIPTION mysub
PUBLICATION insert_only
WITH (enabled = false);
</programlisting></para>
+
+ <para>
+ Create a subscription to a remote server that replicates tables in
+ the <literal>mypub</literal> publication and starts replicating immediately
+ on commit. Pre-existing data is not copied. The application of changes is
+ delayed by 4 hours.
+<programlisting>
+CREATE SUBSCRIPTION mysub
+ CONNECTION 'host=192.0.2.4 port=5432 user=foo dbname=foodb'
+ PUBLICATION mypub
+ WITH (copy_data = false, min_apply_delay = '4h');
+</programlisting></para>
</refsect1>
<refsect1>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index a56ae311c3..c767cc1c3a 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -64,6 +64,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->oid = subid;
sub->dbid = subform->subdbid;
sub->skiplsn = subform->subskiplsn;
+ sub->minapplydelay = subform->subminapplydelay;
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index d2a8c82900..0950b4b74b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1298,9 +1298,10 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
- subbinary, substream, subtwophasestate, subdisableonerr,
- subslotname, subsynccommit, subpublications, suborigin)
+GRANT SELECT (oid, subdbid, subskiplsn, subminapplydelay, subname, subowner,
+ subenabled, subbinary, substream, subtwophasestate,
+ subdisableonerr, subslotname, subsynccommit, subpublications,
+ suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index baff00dd74..bd9b8c0996 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -66,6 +66,7 @@
#define SUBOPT_DISABLE_ON_ERR 0x00000400
#define SUBOPT_LSN 0x00000800
#define SUBOPT_ORIGIN 0x00001000
+#define SUBOPT_MIN_APPLY_DELAY 0x00002000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -90,6 +91,7 @@ typedef struct SubOpts
bool disableonerr;
char *origin;
XLogRecPtr lsn;
+ int min_apply_delay;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -146,6 +148,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->disableonerr = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY))
+ opts->min_apply_delay = 0;
/* Parse options */
foreach(lc, stmt_options)
@@ -324,6 +328,43 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ strcmp(defel->defname, "min_apply_delay") == 0)
+ {
+ char *val,
+ *tmp;
+ Interval *interval;
+ int64 ms;
+
+ if (IsSet(opts->specified_opts, SUBOPT_MIN_APPLY_DELAY))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_MIN_APPLY_DELAY;
+ tmp = defGetString(defel);
+
+ /*
+ * If no unit was specified, then explicitly add 'ms' otherwise
+ * the interval_in function would assume 'seconds'.
+ */
+ if (strspn(tmp, "-0123456789 ") == strlen(tmp))
+ val = psprintf("%sms", tmp);
+ else
+ val = tmp;
+
+ interval = DatumGetIntervalP(DirectFunctionCall3(interval_in,
+ CStringGetDatum(val),
+ ObjectIdGetDatum(InvalidOid),
+ Int32GetDatum(-1)));
+
+ ms = interval2ms(interval);
+ if (ms < 0 || ms > INT_MAX)
+ ereport(ERROR,
+ errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg("%lld ms is outside the valid range for parameter \"%s\" (0 .. %d)",
+ (long long) ms, "min_apply_delay", INT_MAX));
+
+ opts->min_apply_delay = ms;
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -404,6 +445,17 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /* Test the combination of streaming mode and min_apply_delay */
+ if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ opts->min_apply_delay > 0)
+ {
+ if (opts->streaming == LOGICALREP_STREAM_PARALLEL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s and %s are mutually exclusive options",
+ "min_apply_delay > 0", "streaming = parallel"));
+ }
}
/*
@@ -560,7 +612,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
- SUBOPT_DISABLE_ON_ERR | SUBOPT_ORIGIN);
+ SUBOPT_DISABLE_ON_ERR | SUBOPT_ORIGIN |
+ SUBOPT_MIN_APPLY_DELAY);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -625,6 +678,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_oid - 1] = ObjectIdGetDatum(subid);
values[Anum_pg_subscription_subdbid - 1] = ObjectIdGetDatum(MyDatabaseId);
values[Anum_pg_subscription_subskiplsn - 1] = LSNGetDatum(InvalidXLogRecPtr);
+ values[Anum_pg_subscription_subminapplydelay - 1] = Int64GetDatum(opts.min_apply_delay);
values[Anum_pg_subscription_subname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
@@ -1054,7 +1108,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
- SUBOPT_ORIGIN);
+ SUBOPT_ORIGIN | SUBOPT_MIN_APPLY_DELAY);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1098,6 +1152,18 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
if (IsSet(opts.specified_opts, SUBOPT_STREAMING))
{
+ /*
+ * Test the combination of streaming mode and
+ * min_apply_delay
+ */
+ if (opts.streaming == LOGICALREP_STREAM_PARALLEL)
+ if ((IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) && opts.min_apply_delay > 0) ||
+ (!IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) && sub->minapplydelay > 0))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s mode for subscription with %s",
+ "streaming = parallel", "min_apply_delay"));
+
values[Anum_pg_subscription_substream - 1] =
CharGetDatum(opts.streaming);
replaces[Anum_pg_subscription_substream - 1] = true;
@@ -1111,6 +1177,25 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
= true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY))
+ {
+ /*
+ * Test the combination of streaming mode and
+ * min_apply_delay
+ */
+ if (opts.min_apply_delay > 0)
+ if ((IsSet(opts.specified_opts, SUBOPT_STREAMING) && opts.streaming == LOGICALREP_STREAM_PARALLEL) ||
+ (!IsSet(opts.specified_opts, SUBOPT_STREAMING) && sub->stream == LOGICALREP_STREAM_PARALLEL))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s for subscription in %s mode",
+ "min_apply_delay", "streaming = parallel"));
+
+ values[Anum_pg_subscription_subminapplydelay - 1] =
+ Int64GetDatum(opts.min_apply_delay);
+ replaces[Anum_pg_subscription_subminapplydelay - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index 3dfcff2798..cb945243a4 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -704,7 +704,8 @@ pa_process_spooled_messages_if_required(void)
{
apply_spooled_messages(&MyParallelShared->fileset,
MyParallelShared->xid,
- InvalidXLogRecPtr);
+ InvalidXLogRecPtr,
+ 0);
pa_set_fileset_state(MyParallelShared, FS_EMPTY);
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a0084c7ef6..7ef653749b 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -318,6 +318,17 @@ static List *on_commit_wakeup_workers_subids = NIL;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/*
+ * In order to avoid walsender's timeout during time-delayed replication,
+ * it's necessary to keep sending feedback messages during the delay from the
+ * worker process. Meanwhile, the feature delays the apply before starting the
+ * transaction and thus we don't write WALs for the suspended changes during
+ * the wait. Hence, in the case the worker process sends a feedback message
+ * during the delay, we should not make positions of the flushed and apply LSN
+ * overwritten by the last received latest LSN. See send_feedback() for details.
+ */
+static XLogRecPtr last_received = InvalidXLogRecPtr;
+
/* fields valid only when processing streamed transaction */
static bool in_streamed_transaction = false;
@@ -388,10 +399,13 @@ static void stream_write_change(char action, StringInfo s);
static void stream_open_and_write_change(TransactionId xid, char action, StringInfo s);
static void stream_close_file(void);
-static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
+static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply,
+ bool in_delaying_apply);
static void DisableSubscriptionAndExit(void);
+static void maybe_delay_apply(TimestampTz finish_ts);
+
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
@@ -998,6 +1012,99 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
ExecStoreVirtualTuple(slot);
}
+/*
+ * When min_apply_delay parameter is set on the subscriber, we wait long enough
+ * to make sure a transaction is applied at least that interval behind the
+ * publisher.
+ *
+ * While the physical replication applies the delay at commit time, this
+ * feature applies the delay for the next transaction but before starting the
+ * transaction. This is mainly because keeping a transaction that conducted
+ * write operations open for a long time results in some issues such as bloat
+ * and locks.
+ *
+ * The min_apply_delay parameter will take effect only after all tables are in
+ * READY state.
+ *
+ * finish_ts is the commit/prepare time of both regular (non-streamed) and
+ * streamed transactions.
+ */
+static void
+maybe_delay_apply(TimestampTz finish_ts)
+{
+ Assert(finish_ts > 0);
+
+ /* Nothing to do if no delay set */
+ if (MySubscription->minapplydelay <= 0)
+ return;
+
+ /*
+ * The min_apply_delay parameter is ignored until all tablesync workers
+ * have reached READY state. If we allow the delay during the catchup
+ * phase, once we reach the limit of tablesync workers, it will impose a
+ * delay for each subsequent worker. It means it will take a long time to
+ * finish the initial table synchronization.
+ */
+ if (!AllTablesyncsReady())
+ return;
+
+ while (true)
+ {
+ long diffms;
+
+ ResetLatch(MyLatch);
+
+ CHECK_FOR_INTERRUPTS();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /*
+ * Before calculating the time duration, reload the catalog if needed.
+ */
+ if (!in_remote_transaction && !in_streamed_transaction)
+ {
+ AcceptInvalidationMessages();
+ maybe_reread_subscription();
+ }
+
+ diffms = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ TimestampTzPlusMilliseconds(finish_ts, MySubscription->minapplydelay));
+
+ /*
+ * Exit without arming the latch if it's already past time to apply
+ * this transaction.
+ */
+ if (diffms <= 0)
+ break;
+
+ elog(DEBUG2, "logical replication apply delay: %ld ms", diffms);
+
+ /*
+ * Call send_feedback() to prevent the publisher from exiting by
+ * timeout during the delay, when wal_receiver_status_interval is
+ * available.
+ */
+ if (wal_receiver_status_interval > 0
+ && diffms > wal_receiver_status_interval * 1000)
+ {
+ WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ (long) wal_receiver_status_interval * 1000,
+ WAIT_EVENT_RECOVERY_APPLY_DELAY);
+ send_feedback(last_received, true, false, true);
+ }
+ else
+ WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ diffms,
+ WAIT_EVENT_RECOVERY_APPLY_DELAY);
+ }
+}
+
/*
* Handle BEGIN message.
*/
@@ -1012,6 +1119,9 @@ apply_handle_begin(StringInfo s)
logicalrep_read_begin(s, &begin_data);
set_apply_error_context_xact(begin_data.xid, begin_data.final_lsn);
+ /* Should we delay the current transaction? */
+ maybe_delay_apply(begin_data.committime);
+
remote_final_lsn = begin_data.final_lsn;
maybe_start_skipping_changes(begin_data.final_lsn);
@@ -1069,6 +1179,9 @@ apply_handle_begin_prepare(StringInfo s)
logicalrep_read_begin_prepare(s, &begin_data);
set_apply_error_context_xact(begin_data.xid, begin_data.prepare_lsn);
+ /* Should we delay the current prepared transaction? */
+ maybe_delay_apply(begin_data.prepare_time);
+
remote_final_lsn = begin_data.prepare_lsn;
maybe_start_skipping_changes(begin_data.prepare_lsn);
@@ -1316,7 +1429,8 @@ apply_handle_stream_prepare(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset,
- prepare_data.xid, prepare_data.prepare_lsn);
+ prepare_data.xid, prepare_data.prepare_lsn,
+ prepare_data.prepare_time);
/* Mark the transaction as prepared. */
apply_handle_prepare_internal(&prepare_data);
@@ -2010,10 +2124,13 @@ ensure_last_message(FileSet *stream_fileset, TransactionId xid, int fileno,
/*
* Common spoolfile processing.
+ *
+ * The commit/prepare time for streaming transaction is required to achieve
+ * time-delayed replication.
*/
void
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn, TimestampTz finish_ts)
{
StringInfoData s2;
int nchanges;
@@ -2024,6 +2141,21 @@ apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
int fileno;
off_t offset;
+ /*
+ * Should we delay the current transaction?
+ *
+ * Unlike the regular (non-streamed) cases, the delay is applied in a
+ * STREAM COMMIT/STREAM PREPARE message for streamed transactions. The
+ * STREAM START message does not contain a commit/prepare time (it will be
+ * available when the in-progress transaction finishes). Hence, it's not
+ * appropriate to apply a delay at that time.
+ *
+ * It's not allowed to execute time-delayed replication with parallel
+ * apply feature.
+ */
+ if (!am_parallel_apply_worker())
+ maybe_delay_apply(finish_ts);
+
if (!am_parallel_apply_worker())
maybe_start_skipping_changes(lsn);
@@ -2173,7 +2305,7 @@ apply_handle_stream_commit(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset, xid,
- commit_data.commit_lsn);
+ commit_data.commit_lsn, commit_data.committime);
apply_handle_commit_internal(&commit_data);
@@ -3446,7 +3578,7 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
* Apply main loop.
*/
static void
-LogicalRepApplyLoop(XLogRecPtr last_received)
+LogicalRepApplyLoop(void)
{
TimestampTz last_recv_timestamp = GetCurrentTimestamp();
bool ping_sent = false;
@@ -3567,7 +3699,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
if (last_received < end_lsn)
last_received = end_lsn;
- send_feedback(last_received, reply_requested, false);
+ send_feedback(last_received, reply_requested, false, false);
UpdateWorkerStats(last_received, timestamp, true);
}
/* other message types are purposefully ignored */
@@ -3580,7 +3712,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/* confirm all writes so far */
- send_feedback(last_received, false, false);
+ send_feedback(last_received, false, false, false);
if (!in_remote_transaction && !in_streamed_transaction)
{
@@ -3677,7 +3809,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
}
- send_feedback(last_received, requestReply, requestReply);
+ send_feedback(last_received, requestReply, requestReply, false);
/*
* Force reporting to ensure long idle periods don't lead to
@@ -3707,7 +3839,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
* to send a response to avoid timeouts.
*/
static void
-send_feedback(XLogRecPtr recvpos, bool force, bool requestReply)
+send_feedback(XLogRecPtr recvpos, bool force, bool requestReply, bool in_delaying_apply)
{
static StringInfo reply_message = NULL;
static TimestampTz send_time = 0;
@@ -3737,8 +3869,15 @@ send_feedback(XLogRecPtr recvpos, bool force, bool requestReply)
/*
* No outstanding transactions to flush, we can report the latest received
* position. This is important for synchronous replication.
+ *
+ * During the delay of time-delayed replication, do not tell the publisher
+ * that the received latest LSN is already applied and flushed at this
+ * stage, since we don't apply the transaction yet. If we do so, it leads
+ * to a wrong assumption of logical replication progress on the publisher
+ * side. Here, we just send a feedback message to avoid publisher's
+ * timeout during the delay.
*/
- if (!have_pending_txes)
+ if (!have_pending_txes && !in_delaying_apply)
flushpos = writepos = recvpos;
if (writepos < last_writepos)
@@ -4354,11 +4493,11 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname)
* of system resource error and are not repeatable.
*/
static void
-start_apply(XLogRecPtr origin_startpos)
+start_apply(void)
{
PG_TRY();
{
- LogicalRepApplyLoop(origin_startpos);
+ LogicalRepApplyLoop();
}
PG_CATCH();
{
@@ -4645,7 +4784,8 @@ ApplyWorkerMain(Datum main_arg)
}
/* Run the main loop. */
- start_apply(origin_startpos);
+ last_received = origin_startpos;
+ start_apply();
proc_exit(0);
}
diff --git a/src/backend/utils/adt/timestamp.c b/src/backend/utils/adt/timestamp.c
index 928c330897..422e6ad0fa 100644
--- a/src/backend/utils/adt/timestamp.c
+++ b/src/backend/utils/adt/timestamp.c
@@ -2431,6 +2431,35 @@ interval_cmp_internal(const Interval *interval1, const Interval *interval2)
return int128_compare(span1, span2);
}
+/*
+ * Returns the number of milliseconds in the specified Interval.
+ */
+int64
+interval2ms(const Interval *interval)
+{
+ int64 days;
+ int64 ms;
+ int64 result;
+
+ days = interval->month * INT64CONST(30);
+ days += interval->day;
+
+ /* Detect whether the value of interval can cause an overflow */
+ if (pg_mul_s64_overflow(days, MSECS_PER_DAY, &result))
+ ereport(ERROR,
+ errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg("bigint out of range"));
+
+ /* Adds portion time (in ms) to the previous result */
+ ms = interval->time / INT64CONST(1000);
+ if (pg_add_s64_overflow(result, ms, &result))
+ ereport(ERROR,
+ errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg("bigint out of range"));
+
+ return result;
+}
+
Datum
interval_eq(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 527c7651ab..7ffbab0915 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4494,6 +4494,7 @@ getSubscriptions(Archive *fout)
int i_subsynccommit;
int i_subpublications;
int i_subbinary;
+ int i_subminapplydelay;
int i,
ntups;
@@ -4546,9 +4547,14 @@ getSubscriptions(Archive *fout)
LOGICALREP_TWOPHASE_STATE_DISABLED);
if (fout->remoteVersion >= 160000)
- appendPQExpBufferStr(query, " s.suborigin\n");
+ appendPQExpBufferStr(query,
+ " s.suborigin,\n"
+ " s.subminapplydelay\n");
else
- appendPQExpBuffer(query, " '%s' AS suborigin\n", LOGICALREP_ORIGIN_ANY);
+ {
+ appendPQExpBuffer(query, " '%s' AS suborigin,\n", LOGICALREP_ORIGIN_ANY);
+ appendPQExpBufferStr(query, " 0 AS subminapplydelay\n");
+ }
appendPQExpBufferStr(query,
"FROM pg_subscription s\n"
@@ -4576,6 +4582,7 @@ getSubscriptions(Archive *fout)
i_subtwophasestate = PQfnumber(res, "subtwophasestate");
i_subdisableonerr = PQfnumber(res, "subdisableonerr");
i_suborigin = PQfnumber(res, "suborigin");
+ i_subminapplydelay = PQfnumber(res, "subminapplydelay");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4606,6 +4613,8 @@ getSubscriptions(Archive *fout)
subinfo[i].subdisableonerr =
pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
subinfo[i].suborigin = pg_strdup(PQgetvalue(res, i, i_suborigin));
+ subinfo[i].subminapplydelay =
+ strtoi64(PQgetvalue(res, i, i_subminapplydelay), NULL, 10);
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -4687,6 +4696,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
+ if (subinfo->subminapplydelay > 0)
+ appendPQExpBuffer(query, ", min_apply_delay = '" INT64_FORMAT " ms'", subinfo->subminapplydelay);
+
appendPQExpBufferStr(query, ");\n");
if (subinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index e7cbd8d7ed..e2525f70ab 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -661,6 +661,7 @@ typedef struct _SubscriptionInfo
char *subdisableonerr;
char *suborigin;
char *subsynccommit;
+ int64 subminapplydelay;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index c8a0bb7b3a..8a27063bed 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6472,7 +6472,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false, false, false, false};
+ false, false, false, false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6527,10 +6527,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Two-phase commit"),
gettext_noop("Disable on error"));
+ /* Origin and min_apply_delay are only supported in v16 and higher */
if (pset.sversion >= 160000)
appendPQExpBuffer(&buf,
- ", suborigin AS \"%s\"\n",
- gettext_noop("Origin"));
+ ", suborigin AS \"%s\"\n"
+ ", subminapplydelay AS \"%s\"\n",
+ gettext_noop("Origin"),
+ gettext_noop("Min apply delay (ms)"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 5e1882eaea..e8b9a43a47 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1925,7 +1925,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "origin", "slot_name",
+ COMPLETE_WITH("binary", "disable_on_error", "min_apply_delay", "origin", "slot_name",
"streaming", "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
@@ -3268,7 +3268,7 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "origin", "slot_name",
+ "disable_on_error", "enabled", "min_apply_delay", "origin", "slot_name",
"streaming", "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index b0f2a1705d..078495cbf0 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -70,6 +70,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
XLogRecPtr subskiplsn; /* All changes finished at this LSN are
* skipped */
+ int64 subminapplydelay; /* Replication apply delay */
+
NameData subname; /* Name of the subscription */
Oid subowner BKI_LOOKUP(pg_authid); /* Owner of the subscription */
@@ -120,6 +122,7 @@ typedef struct Subscription
* in */
XLogRecPtr skiplsn; /* All changes finished at this LSN are
* skipped */
+ int64 minapplydelay; /* Replication apply delay */
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
diff --git a/src/include/datatype/timestamp.h b/src/include/datatype/timestamp.h
index 21a37e21e9..8b368af299 100644
--- a/src/include/datatype/timestamp.h
+++ b/src/include/datatype/timestamp.h
@@ -127,6 +127,8 @@ struct pg_itm_in
#define SECS_PER_MINUTE 60
#define MINS_PER_HOUR 60
+#define MSECS_PER_DAY INT64CONST(86400000)
+
#define USECS_PER_DAY INT64CONST(86400000000)
#define USECS_PER_HOUR INT64CONST(3600000000)
#define USECS_PER_MINUTE INT64CONST(60000000)
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index db891eea8a..4ef132d243 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -255,7 +255,7 @@ extern void stream_stop_internal(TransactionId xid);
/* Common streaming function to apply all the spooled messages */
extern void apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn);
+ XLogRecPtr lsn, TimestampTz finish_ts);
extern void apply_dispatch(StringInfo s);
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index 42f802bb9d..534051fe13 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -102,6 +102,8 @@ extern bool TimestampDifferenceExceeds(TimestampTz start_time,
TimestampTz stop_time,
int msec);
+extern int64 interval2ms(const Interval *interval);
+
extern TimestampTz time_t_to_timestamptz(pg_time_t tm);
extern pg_time_t timestamptz_to_time_t(TimestampTz t);
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 4e5cb0d3a9..dbf0803568 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -114,18 +114,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -135,10 +135,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -155,10 +155,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -167,10 +167,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -202,10 +202,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -239,19 +239,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -263,27 +263,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -298,10 +298,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -316,10 +316,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -355,10 +355,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -367,10 +367,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -380,10 +380,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,20 +396,61 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail -- min_apply_delay must be a non-negative integer
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = foo);
+ERROR: invalid input syntax for type interval: "foo"
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = -1);
+ERROR: -1 ms is outside the valid range for parameter "min_apply_delay" (0 .. 2147483647)
+-- fail - utilizing streaming = parallel with time delayed replication is not supported.
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = parallel, min_apply_delay = '1 day');
+ERROR: min_apply_delay > 0 and streaming = parallel are mutually exclusive options
+-- success -- value without unit is taken as milliseconds
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = 123);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 123 | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+-- success -- interval is converted into ms and stored as integer
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = '4h 27min 35s');
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 16055000 | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+-- fail - alter subscription with streaming = parallel should fail when time delayed replication is set.
+ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
+ERROR: cannot enable streaming = parallel mode for subscription with min_apply_delay
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - alter subscription with min_apply_delay should fail when streaming = parallel is set.
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = parallel);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 123);
+ERROR: cannot enable min_apply_delay for subscription in streaming = parallel mode
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
RESET SESSION AUTHORIZATION;
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 5f27b7d776..4b90b85702 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -279,6 +279,31 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail -- min_apply_delay must be a non-negative integer
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = foo);
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = -1);
+-- fail - utilizing streaming = parallel with time delayed replication is not supported.
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = parallel, min_apply_delay = '1 day');
+
+-- success -- value without unit is taken as milliseconds
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = 123);
+\dRs+
+
+-- success -- interval is converted into ms and stored as integer
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = '4h 27min 35s');
+\dRs+
+
+-- fail - alter subscription with streaming = parallel should fail when time delayed replication is set.
+ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+-- fail - alter subscription with min_apply_delay should fail when streaming = parallel is set.
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = parallel);
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 123);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index 3db0fdfd96..a186876eb4 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -38,6 +38,7 @@ tests += {
't/029_on_error.pl',
't/030_origin.pl',
't/031_column_list.pl',
+ 't/032_apply_delay.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/032_apply_delay.pl b/src/test/subscription/t/032_apply_delay.pl
new file mode 100644
index 0000000000..25459e3dec
--- /dev/null
+++ b/src/test/subscription/t/032_apply_delay.pl
@@ -0,0 +1,182 @@
+
+# Copyright (c) 2023, PostgreSQL Global Development Group
+
+# Test replication apply delay
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $log_location = 0;
+
+# Confirm the time-delayed replication has been effective from the server log
+# message where the apply worker emits for applying delay. When necessary,
+# verifies that the current worker's delayed time is sufficiently bigger than
+# the expected value, in order to check any update of the min_apply_delay.
+sub check_apply_delay_log
+{
+ my ($node_subscriber, $message, $expected) = @_;
+ $expected = 0 unless defined $expected;
+
+ my $old_log_location = $log_location;
+
+ $log_location = $node_subscriber->wait_for_log(qr/$message/, $log_location);
+
+ cmp_ok($log_location, '>', $old_log_location,
+ "logfile contains triggered logical replication apply delay"
+ );
+
+ if ($expected > 0)
+ {
+ # Get the delay time in the server log
+ my $contents = slurp_file($node_subscriber->logfile, $old_log_location);
+ $contents =~
+ qr/$message: (\d+) ms/
+ or die "could not get delayed time";
+ my $logged_delay = $1;
+
+ # Is it larger than expected?
+ cmp_ok($logged_delay, '>', $expected,
+ "The wait time of the apply worker is long enough expectedly"
+ );
+ }
+}
+
+sub check_apply_delay_time
+{
+ my ($node_publisher, $node_subscriber, $primary_key, $expected_diffs) = @_;
+
+ my $inserted_time_on_pub = $node_publisher->safe_psql('postgres', qq[
+ SELECT extract(epoch from c) FROM test_tab WHERE a = $primary_key;
+ ]);
+
+ my $inserted_time_on_sub = $node_subscriber->safe_psql('postgres', qq[
+ SELECT extract(epoch from c) FROM test_tab WHERE a = $primary_key;
+ ]);
+
+ cmp_ok($inserted_time_on_sub - $inserted_time_on_pub, '>', $expected_diffs,
+ "The tuple on the subscriber was modified later than the publisher");
+}
+
+# Create publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = debug2");
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar, c timestamptz DEFAULT now())");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)"
+);
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+# The column c must not be published because we want to compare the time
+# difference.
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE test_tab (a, b)");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on, min_apply_delay = '3s')"
+);
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Check log starting now for logical replication apply delay
+$log_location = -s $node_subscriber->logfile;
+
+my $result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM test_tab");
+is($result, qq(2|1|2), 'check initial data was copied to subscriber');
+
+# New row to trigger apply delay
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (3, 'baz')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (4, 'abc')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (5, 'def')");
+
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM test_tab");
+is($result, qq(5|1|5), 'check if the new rows were applied to subscriber');
+
+check_apply_delay_log($node_subscriber, "logical replication apply delay");
+check_apply_delay_time($node_publisher, $node_subscriber, '5', '3');
+
+# Reduce the amounts of writes for spooling file
+$node_publisher->append_conf('postgres.conf',
+ 'logical_decoding_mode = immediate');
+$node_publisher->reload;
+
+# Test streamed transaction by insert, update and delete enough rows to exceed
+# 64kB limit.
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6, 8) s(i);");
+
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM test_tab");
+is($result, qq(8|1|8), 'check if the new rows were applied to subscriber');
+
+check_apply_delay_log($node_subscriber, "logical replication apply delay");
+check_apply_delay_time($node_publisher, $node_subscriber, '8', '3');
+
+# Test whether ALTER SUBSCRIPTION changes the delayed time of the apply worker
+# (1 day 1 minute).
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (min_apply_delay = 86460000)"
+);
+
+# New row to trigger apply delay
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (0, 'foobar')");
+
+# Make sure we have long enough min_apply_delay after the ALTER command
+check_apply_delay_log($node_subscriber, "logical replication apply delay", "80000000");
+
+# Disable subscription and the worker should die immediately
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub DISABLE;"
+);
+
+# Wait until worker dies
+my $sub_query =
+ "SELECT count(1) = 0 FROM pg_stat_subscription WHERE subname = 'tap_sub' AND pid IS NOT NULL;";
+$node_subscriber->poll_query_until('postgres', $sub_query)
+ or die "Timed out while waiting for subscriber to die";
+
+# Confirm the suspended record doesn't get applied by the ALTER DISABLE
+# command.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(a) FROM test_tab WHERE a = 0;");
+is($result, qq(0), "check if the delayed transaction doesn't get applied expectedly");
+
+$node_subscriber->stop;
+$node_publisher->stop;
+
+done_testing();
--
2.30.0
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-17 12:45 Hayato Kuroda (Fujitsu) <[email protected]>
parent: Takamichi Osumi (Fujitsu) <[email protected]>
1 sibling, 0 replies; 40+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2023-01-17 12:45 UTC (permalink / raw)
To: Takamichi Osumi (Fujitsu) <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; 'vignesh C' <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>
Dear hackers,
> At present, in the latter case,
> the apply worker exits without applying the suspended transaction
> after ALTER SUBSCRIPTION DISABLE command for the subscription.
> Meanwhile, there is no "disabling" command for physical replication,
> but I checked the behavior about what happens for promoting a secondary
> during the delay of recovery_min_apply_delay for physical replication as one
> example.
> The transaction has become visible even in the promoting in the middle of delay.
>
> I'm not sure if I should make the time-delayed LR aligned with this behavior.
> Does someone has an opinion for this ?
I put my opinion here. The current specification is correct; we should not follow
a physical replication manner.
One motivation for this feature is to offer opportunities to correct data loss
errors. When accidental delete events occur, DBA can stop propagations on subscribers
by disabling the subscription, with the patch at present.
IIUC, when the subscription is disabled before transactions are started,
workers exit and stop applications. This feature delays starting txns, so we
should regard such an alternation as that is executed before the transaction.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-18 07:06 Peter Smith <[email protected]>
parent: Takamichi Osumi (Fujitsu) <[email protected]>
1 sibling, 3 replies; 40+ messages in thread
From: Peter Smith @ 2023-01-18 07:06 UTC (permalink / raw)
To: Takamichi Osumi (Fujitsu) <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
Here are my review comments for the latest patch v16-0001. (excluding
the test code)
======
General
1.
Since the value of min_apply_delay cannot be < 0, I was thinking
probably it should have been declared everywhere in this patch as a
uint64 instead of an int64, right?
======
Commit message
2.
If the subscription sets min_apply_delay parameter, the logical
replication worker will delay the transaction commit for min_apply_delay
milliseconds.
~
IMO there should be another sentence before this just to say that a
new parameter is being added:
e.g.
This patch implements a new subscription parameter called 'min_apply_delay'.
======
doc/src/sgml/config.sgml
3.
+ <para>
+ For time-delayed logical replication, the apply worker sends a Standby
+ Status Update message to the corresponding publisher per the indicated
+ time of this parameter. Therefore, if this parameter is longer than
+ <literal>wal_sender_timeout</literal> on the publisher, then the
+ walsender doesn't get any update message during the delay and repeatedly
+ terminates due to the timeout errors. Hence, make sure this parameter is
+ shorter than the <literal>wal_sender_timeout</literal> of the publisher.
+ If this parameter is set to zero with time-delayed replication, the
+ apply worker doesn't send any feedback messages during the
+ <literal>min_apply_delay</literal>.
+ </para>
This paragraph seemed confusing. I think it needs to be reworded to
change all of the "this parameter" references because there are at
least 3 different parameters mentioned in this paragraph. e.g. maybe
just change them to explicitly name the parameter you are talking
about.
I also think it needs to mention the ‘min_apply_delay’ subscription
parameter up-front and then refer to it appropriately.
The end result might be something like I wrote below (this is just my
guess – probably you can word it better).
SUGGESTION
For time-delayed logical replication (i.e. when the subscription is
created with parameter min_apply_delay > 0), the apply worker sends a
Standby Status Update message to the publisher with a period of
wal_receiver_status_interval . Make sure to set
wal_receiver_status_interval less than the wal_sender_timeout on the
publisher, otherwise, the walsender will repeatedly terminate due to
the timeout errors. If wal_receiver_status_interval is set to zero,
the apply worker doesn't send any feedback messages during the
subscriber’s min_apply_delay period.
======
doc/src/sgml/ref/create_subscription.sgml
4.
+ <para>
+ By default, the subscriber applies changes as soon as possible. As
+ with the physical replication feature
+ (<xref linkend="guc-recovery-min-apply-delay"/>), it can be useful to
+ have a time-delayed logical replica. This parameter lets the user to
+ delay the application of changes by a specified amount of
time. If this
+ value is specified without units, it is taken as milliseconds. The
+ default is zero(no delay).
+ </para>
4a.
As with the physical replication feature (recovery_min_apply_delay),
it can be useful to have a time-delayed logical replica.
IMO not sure that the above sentence is necessary. It seems only to be
saying that this parameter can be useful. Why do we need to say that?
~
4b.
"This parameter lets the user to delay" -> "This parameter lets the user delay"
OR
"This parameter lets the user to delay" -> "This parameter allows the
user to delay"
~
4c.
"If this value is specified without units" -> "If the value is
specified without units"
~
4d.
"zero(no delay)." -> "zero (no delay)."
----
5.
+ <para>
+ The delay occurs only on WAL records for transaction begins and after
+ the initial table synchronization. It is possible that the
+ replication delay between publisher and subscriber exceeds the value
+ of this parameter, in which case no delay is added. Note that the
+ delay is calculated between the WAL time stamp as written on
+ publisher and the current time on the subscriber. Time
spent in logical
+ decoding and in transferring the transaction may reduce the
actual wait
+ time. If the system clocks on publisher and subscriber are not
+ synchronized, this may lead to apply changes earlier than expected,
+ but this is not a major issue because this parameter is
typically much
+ larger than the time deviations between servers. Note that if this
+ parameter is set to a long delay, the replication will stop if the
+ replication slot falls behind the current LSN by more than
+ <link
linkend="guc-max-slot-wal-keep-size"><literal>max_slot_wal_keep_size</literal></link>.
+ </para>
I think the first part can be reworded slightly. See what you think
about the suggestion below.
SUGGESTION
Any delay occurs only on WAL records for transaction begins after all
initial table synchronization has finished. The delay is calculated
between the WAL timestamp as written on the publisher and the current
time on the subscriber. Any overhead of time spent in logical decoding
and in transferring the transaction may reduce the actual wait time.
It is also possible that the overhead already exceeds the requested
'min_apply_delay' value, in which case no additional wait is
necessary. If the system clocks...
----
6.
+ <para>
+ Setting streaming to <literal>parallel</literal> mode and
<literal>min_apply_delay</literal>
+ simultaneously is not supported.
+ </para>
SUGGESTION
A non-zero min_apply_delay parameter is not allowed when streaming in
parallel mode.
======
src/backend/commands/subscriptioncmds.c
7. parse_subscription_options
@@ -404,6 +445,17 @@ parse_subscription_options(ParseState *pstate,
List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /* Test the combination of streaming mode and min_apply_delay */
+ if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ opts->min_apply_delay > 0)
+ {
+ if (opts->streaming == LOGICALREP_STREAM_PARALLEL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s and %s are mutually exclusive options",
+ "min_apply_delay > 0", "streaming = parallel"));
+ }
SUGGESTION (comment)
The combination of parallel streaming mode and min_apply_delay is not allowed.
~~~
8. AlterSubscription (general)
I observed during testing there are 3 different errors….
At subscription CREATE time you can get this error:
ERROR: min_apply_delay > 0 and streaming = parallel are mutually
exclusive options
If you try to ALTER the min_apply_delay when already streaming =
parallel you can get this error:
ERROR: cannot enable min_apply_delay for subscription in streaming =
parallel mode
If you try to ALTER the streaming to be parallel if there is already a
min_apply_delay > 0 then you can get this error:
ERROR: cannot enable streaming = parallel mode for subscription with
min_apply_delay
~
IMO there is no need to have 3 different error message texts. I think
all these cases are explained by just the first text (ERROR:
min_apply_delay > 0 and streaming = parallel are mutually exclusive
options)
~~~
9. AlterSubscription
@@ -1098,6 +1152,18 @@ AlterSubscription(ParseState *pstate,
AlterSubscriptionStmt *stmt,
if (IsSet(opts.specified_opts, SUBOPT_STREAMING))
{
+ /*
+ * Test the combination of streaming mode and
+ * min_apply_delay
+ */
+ if (opts.streaming == LOGICALREP_STREAM_PARALLEL)
+ if ((IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) &&
opts.min_apply_delay > 0) ||
+ (!IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) &&
sub->minapplydelay > 0))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s mode for subscription with %s",
+ "streaming = parallel", "min_apply_delay"));
+
9a.
SUGGESTION (comment)
The combination of parallel streaming mode and min_apply_delay is not allowed.
~
9b.
(see AlterSubscription general review comment #8 above)
Here you can use the same comment error message that says
min_apply_delay > 0 and streaming = parallel are mutually exclusive
options.
~~~
10. AlterSubscription
@@ -1111,6 +1177,25 @@ AlterSubscription(ParseState *pstate,
AlterSubscriptionStmt *stmt,
= true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY))
+ {
+ /*
+ * Test the combination of streaming mode and
+ * min_apply_delay
+ */
+ if (opts.min_apply_delay > 0)
+ if ((IsSet(opts.specified_opts, SUBOPT_STREAMING) && opts.streaming
== LOGICALREP_STREAM_PARALLEL) ||
+ (!IsSet(opts.specified_opts, SUBOPT_STREAMING) && sub->stream ==
LOGICALREP_STREAM_PARALLEL))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s for subscription in %s mode",
+ "min_apply_delay", "streaming = parallel"));
+
+ values[Anum_pg_subscription_subminapplydelay - 1] =
+ Int64GetDatum(opts.min_apply_delay);
+ replaces[Anum_pg_subscription_subminapplydelay - 1] = true;
+ }
10a.
SUGGESTION (comment)
The combination of parallel streaming mode and min_apply_delay is not allowed.
~
10b.
(see AlterSubscription general review comment #8 above)
Here you can use the same comment error message that says
min_apply_delay > 0 and streaming = parallel are mutually exclusive
options.
======
.../replication/logical/applyparallelworker.c
11.
@@ -704,7 +704,8 @@ pa_process_spooled_messages_if_required(void)
{
apply_spooled_messages(&MyParallelShared->fileset,
MyParallelShared->xid,
- InvalidXLogRecPtr);
+ InvalidXLogRecPtr,
+ 0);
IMO this passing of 0 is a bit strange because it is currently acting
like a dummy value since the apply_spooled_messages will never make
use of the 'finish_ts' anyway (since this call is from a parallel
apply worker).
I think a better way to code this might be to pass the 0 (same as you
are doing here) but inside the apply_spooled_messages change the code:
FROM
if (!am_parallel_apply_worker())
maybe_delay_apply(finish_ts);
TO
if (finish_ts)
maybe_delay_apply(finish_ts);
That does 2 things.
- It makes the passed-in 0 have some meaning
- It simplifies the apply_spooled_messages code
======
src/backend/replication/logical/worker.c
12.
@@ -318,6 +318,17 @@ static List *on_commit_wakeup_workers_subids = NIL;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/*
+ * In order to avoid walsender's timeout during time-delayed replication,
+ * it's necessary to keep sending feedback messages during the delay from the
+ * worker process. Meanwhile, the feature delays the apply before starting the
+ * transaction and thus we don't write WALs for the suspended changes during
+ * the wait. Hence, in the case the worker process sends a feedback message
+ * during the delay, we should not make positions of the flushed and apply LSN
+ * overwritten by the last received latest LSN. See send_feedback()
for details.
+ */
+static XLogRecPtr last_received = InvalidXLogRecPtr;
12a.
Suggest a small change to the first sentence of the comment.
BEFORE
In order to avoid walsender's timeout during time-delayed replication,
it's necessary to keep sending feedback messages during the delay from
the worker process.
AFTER
In order to avoid walsender timeout for time-delayed replication the
worker process keeps sending feedback messages during the delay
period.
~
12b.
"Hence, in the case" -> "When"
~~~
13. forward declare
-static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
+static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply,
+ bool in_delaying_apply);
Change the param name:
"in_delaying_apply" -> "in_delayed_apply” (??)
~~~
14. maybe_delay_apply
+ /* Nothing to do if no delay set */
+ if (MySubscription->minapplydelay <= 0)
+ return;
IIUC min_apply_delay cannot be < 0 so this condition could simply be:
if (!MySubscription->minapplydelay)
return;
~~~
15. maybe_delay_apply
+ /*
+ * The min_apply_delay parameter is ignored until all tablesync workers
+ * have reached READY state. If we allow the delay during the catchup
+ * phase, once we reach the limit of tablesync workers, it will impose a
+ * delay for each subsequent worker. It means it will take a long time to
+ * finish the initial table synchronization.
+ */
+ if (!AllTablesyncsReady())
+ return;
SUGGESTION (slight rewording)
The min_apply_delay parameter is ignored until all tablesync workers
have reached READY state. This is because if we allowed the delay
during the catchup phase, then once we reached the limit of tablesync
workers it would impose a delay for each subsequent worker. That would
cause initial table synchronization completion to take a long time.
~~~
16. maybe_delay_apply
+ while (true)
+ {
+ long diffms;
+
+ ResetLatch(MyLatch);
+
+ CHECK_FOR_INTERRUPTS();
IMO there should be some small explanatory comment here at the top of
the while loop.
~~~
17. apply_spooled_messages
@@ -2024,6 +2141,21 @@ apply_spooled_messages(FileSet *stream_fileset,
TransactionId xid,
int fileno;
off_t offset;
+ /*
+ * Should we delay the current transaction?
+ *
+ * Unlike the regular (non-streamed) cases, the delay is applied in a
+ * STREAM COMMIT/STREAM PREPARE message for streamed transactions. The
+ * STREAM START message does not contain a commit/prepare time (it will be
+ * available when the in-progress transaction finishes). Hence, it's not
+ * appropriate to apply a delay at that time.
+ *
+ * It's not allowed to execute time-delayed replication with parallel
+ * apply feature.
+ */
+ if (!am_parallel_apply_worker())
+ maybe_delay_apply(finish_ts);
That whole comment part "Unlike the regular (non-streamed) cases"
seems misplaced here. Perhaps this part of the comment is better put
into the function header where the meaning of 'finish_ts' is
explained?
~~~
18. apply_spooled_messages
+ * It's not allowed to execute time-delayed replication with parallel
+ * apply feature.
+ */
+ if (!am_parallel_apply_worker())
+ maybe_delay_apply(finish_ts);
As was mentioned in comment #11 above this code could be changed like
if (finish_ts)
maybe_delay_apply(finish_ts);
then you don't even need to make mention of "parallel apply" at all here.
OTOH if you want to still have the parallel apply comment then maybe
reword it like this:
"It is not allowed to combine time-delayed replication with the
parallel apply feature."
~~~
19. apply_spooled_messages
If you chose not to do my suggestion from comment #11, then there are
2 identical conditions (!am_parallel_apply_worker()); In this case, I
was wondering if it would be better to refactor to use a single
condition instead.
~~~
20. send_feedback
(same as comment #13)
Maybe change the new param name to “in_delayed_apply”?
~~~
21.
@@ -3737,8 +3869,15 @@ send_feedback(XLogRecPtr recvpos, bool force,
bool requestReply)
/*
* No outstanding transactions to flush, we can report the latest received
* position. This is important for synchronous replication.
+ *
+ * During the delay of time-delayed replication, do not tell the publisher
+ * that the received latest LSN is already applied and flushed at this
+ * stage, since we don't apply the transaction yet. If we do so, it leads
+ * to a wrong assumption of logical replication progress on the publisher
+ * side. Here, we just send a feedback message to avoid publisher's
+ * timeout during the delay.
*/
Minor rewording of the comment
SUGGESTION
If the subscriber side apply is delayed (because of time-delayed
replication) then do not tell the publisher that the received latest
LSN is already applied and flushed, otherwise, it leads to the
publisher side making a wrong assumption of logical replication
progress. Instead, we just send a feedback message to avoid a
publisher timeout during the delay.
======
src/bin/pg_dump/pg_dump.c
22.
@@ -4546,9 +4547,14 @@ getSubscriptions(Archive *fout)
LOGICALREP_TWOPHASE_STATE_DISABLED);
if (fout->remoteVersion >= 160000)
- appendPQExpBufferStr(query, " s.suborigin\n");
+ appendPQExpBufferStr(query,
+ " s.suborigin,\n"
+ " s.subminapplydelay\n");
else
- appendPQExpBuffer(query, " '%s' AS suborigin\n", LOGICALREP_ORIGIN_ANY);
+ {
+ appendPQExpBuffer(query, " '%s' AS suborigin,\n", LOGICALREP_ORIGIN_ANY);
+ appendPQExpBufferStr(query, " 0 AS subminapplydelay\n");
+ }
Can’t those appends in the else part can be combined to a single
appendPQExpBuffer
appendPQExpBuffer(query,
" '%s' AS suborigin,\n"
" 0 AS subminapplydelay\n"
LOGICALREP_ORIGIN_ANY);
======
src/include/catalog/pg_subscription.h
23.
@@ -70,6 +70,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
BKI_SHARED_RELATION BKI_ROW
XLogRecPtr subskiplsn; /* All changes finished at this LSN are
* skipped */
+ int64 subminapplydelay; /* Replication apply delay */
+
NameData subname; /* Name of the subscription */
Oid subowner BKI_LOOKUP(pg_authid); /* Owner of the subscription */
SUGGESTION (for comment)
Replication apply delay (ms)
~~
24.
@@ -120,6 +122,7 @@ typedef struct Subscription
* in */
XLogRecPtr skiplsn; /* All changes finished at this LSN are
* skipped */
+ int64 minapplydelay; /* Replication apply delay */
SUGGESTION (for comment)
Replication apply delay (ms)
------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-19 01:41 Peter Smith <[email protected]>
parent: Peter Smith <[email protected]>
2 siblings, 1 reply; 40+ messages in thread
From: Peter Smith @ 2023-01-19 01:41 UTC (permalink / raw)
To: Takamichi Osumi (Fujitsu) <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Wed, Jan 18, 2023 at 6:06 PM Peter Smith <[email protected]> wrote:
>
> Here are my review comments for the latest patch v16-0001. (excluding
> the test code)
>
...
>
> 8. AlterSubscription (general)
>
> I observed during testing there are 3 different errors….
>
> At subscription CREATE time you can get this error:
> ERROR: min_apply_delay > 0 and streaming = parallel are mutually
> exclusive options
>
> If you try to ALTER the min_apply_delay when already streaming =
> parallel you can get this error:
> ERROR: cannot enable min_apply_delay for subscription in streaming =
> parallel mode
>
> If you try to ALTER the streaming to be parallel if there is already a
> min_apply_delay > 0 then you can get this error:
> ERROR: cannot enable streaming = parallel mode for subscription with
> min_apply_delay
>
> ~
>
> IMO there is no need to have 3 different error message texts. I think
> all these cases are explained by just the first text (ERROR:
> min_apply_delay > 0 and streaming = parallel are mutually exclusive
> options)
>
>
After checking the regression test output I can see the merit of your
separate error messages like this, even if they are maybe not strictly
necessary. So feel free to ignore my previous review comment.
------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-19 01:49 Peter Smith <[email protected]>
parent: Peter Smith <[email protected]>
2 siblings, 1 reply; 40+ messages in thread
From: Peter Smith @ 2023-01-19 01:49 UTC (permalink / raw)
To: Takamichi Osumi (Fujitsu) <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Wed, Jan 18, 2023 at 6:06 PM Peter Smith <[email protected]> wrote:
>
> Here are my review comments for the latest patch v16-0001. (excluding
> the test code)
>
And here are some review comments for the v16-0001 test code.
======
src/test/regress/sql/subscription.sql
1. General
For all comments
"time delayed replication" -> "time-delayed replication" maybe is better?
~~~
2.
-- fail - utilizing streaming = parallel with time delayed replication
is not supported.
For readability please put a blank line before this test.
~~~
3.
-- success -- value without unit is taken as milliseconds
"value" -> "min_apply_delay value"
~~~
4.
-- success -- interval is converted into ms and stored as integer
"interval" -> "min_apply_delay interval"
"integer" -> "an integer"
~~~
5.
You could also add another test where min_apply_delay is 0
Then the following combination can be confirmed OK -- success create
subscription with (streaming=parallel, min_apply_delay=0)
~~
6.
-- fail - alter subscription with min_apply_delay should fail when
streaming = parallel is set.
CREATE SUBSCRIPTION regress_testsub CONNECTION
'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect =
false, streaming = parallel);
There is another way to do this test without creating a brand-new
subscription. You could just alter the existing subscription like:
ALTER ... SET (min_apply_delay = 0)
then ALTER ... SET (parallel = streaming)
then ALTER ... SET (min_apply_delay = 123)
======
src/test/subscription/t/032_apply_delay.pl
7. sub check_apply_delay_log
my ($node_subscriber, $message, $expected) = @_;
Why pass in the message text? I is always the same so can be hardwired
in this function, right?
~~~
8.
# Get the delay time in the server log
"int the server log" -> "from the server log" (?)
~~~
9.
qr/$message: (\d+) ms/
or die "could not get delayed time";
my $logged_delay = $1;
# Is it larger than expected?
cmp_ok($logged_delay, '>', $expected,
"The wait time of the apply worker is long enough expectedly"
);
9a.
"could not get delayed time" -> "could not get the apply worker wait time"
9b.
"The wait time of the apply worker is long enough expectedly" -> "The
apply worker wait time has expected duration"
~~~
10.
sub check_apply_delay_time
Maybe a brief explanatory comment for this function is needed to
explain the unreplicated column c.
~~~
11.
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr
application_name=$appname' PUBLICATION tap_pub WITH (streaming = on,
min_apply_delay = '3s')"
I think there should be a comment here highlighting that you are
setting up a subscriber time delay of 3 seconds, and then later you
can better describe the parameters for the checking functions...
e.g. (add this comment)
# verifies that the subscriber lags the publisher by at least 3 seconds
check_apply_delay_time($node_publisher, $node_subscriber, '5', '3');
e.g.
# verifies that the subscriber lags the publisher by at least 3 seconds
check_apply_delay_time($node_publisher, $node_subscriber, '8', '3');
~~~
12.
# Test whether ALTER SUBSCRIPTION changes the delayed time of the apply worker
# (1 day 1 minute).
$node_subscriber->safe_psql('postgres',
"ALTER SUBSCRIPTION tap_sub SET (min_apply_delay = 86460000)"
);
Update the comment with another note.
# Note - The extra 1 min is to account for any decoding/network overhead.
~~~
13.
# Make sure we have long enough min_apply_delay after the ALTER command
check_apply_delay_log($node_subscriber, "logical replication apply
delay", "80000000");
IMO the expectation of 1 day (86460000 ms) wait time might be a better
number for your "expected" value.
So update the comment/call like this:
# Make sure the apply worker knows to wait for more than 1 day (86400000 ms)
check_apply_delay_log($node_subscriber, "logical replication apply
delay", "86400000");
------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-19 06:35 Takamichi Osumi (Fujitsu) <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 3 replies; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-19 06:35 UTC (permalink / raw)
To: 'Peter Smith' <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Thursday, January 19, 2023 10:49 AM Peter Smith <[email protected]> wrote:
> On Wed, Jan 18, 2023 at 6:06 PM Peter Smith <[email protected]>
> wrote:
> >
> > Here are my review comments for the latest patch v16-0001. (excluding
> > the test code)
> >
>
> And here are some review comments for the v16-0001 test code.
Hi, thanks for your review !
> ======
>
> src/test/regress/sql/subscription.sql
>
> 1. General
> For all comments
>
> "time delayed replication" -> "time-delayed replication" maybe is better?
Fixed.
> ~~~
>
> 2.
> -- fail - utilizing streaming = parallel with time delayed replication is not
> supported.
>
> For readability please put a blank line before this test.
Fixed.
> ~~~
>
> 3.
> -- success -- value without unit is taken as milliseconds
>
> "value" -> "min_apply_delay value"
Fixed.
> ~~~
>
> 4.
> -- success -- interval is converted into ms and stored as integer
>
> "interval" -> "min_apply_delay interval"
>
> "integer" -> "an integer"
Both are fixed.
> ~~~
>
> 5.
> You could also add another test where min_apply_delay is 0
>
> Then the following combination can be confirmed OK -- success create
> subscription with (streaming=parallel, min_apply_delay=0)
This combination is added with the modification for #6.
> ~~
>
> 6.
> -- fail - alter subscription with min_apply_delay should fail when streaming =
> parallel is set.
> CREATE SUBSCRIPTION regress_testsub CONNECTION
> 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false,
> streaming = parallel);
>
> There is another way to do this test without creating a brand-new subscription.
> You could just alter the existing subscription like:
> ALTER ... SET (min_apply_delay = 0)
> then ALTER ... SET (parallel = streaming) then ALTER ... SET (min_apply_delay
> = 123)
Fixed.
> ======
>
> src/test/subscription/t/032_apply_delay.pl
>
> 7. sub check_apply_delay_log
>
> my ($node_subscriber, $message, $expected) = @_;
>
> Why pass in the message text? I is always the same so can be hardwired in this
> function, right?
Fixed.
> ~~~
>
> 8.
> # Get the delay time in the server log
>
> "int the server log" -> "from the server log" (?)
Fixed.
> ~~~
>
> 9.
> qr/$message: (\d+) ms/
> or die "could not get delayed time";
> my $logged_delay = $1;
>
> # Is it larger than expected?
> cmp_ok($logged_delay, '>', $expected,
> "The wait time of the apply worker is long enough expectedly"
> );
>
> 9a.
> "could not get delayed time" -> "could not get the apply worker wait time"
>
> 9b.
> "The wait time of the apply worker is long enough expectedly" -> "The apply
> worker wait time has expected duration"
Both are fixed.
> ~~~
>
> 10.
> sub check_apply_delay_time
>
>
> Maybe a brief explanatory comment for this function is needed to explain the
> unreplicated column c.
Added.
> ~~~
>
> 11.
> $node_subscriber->safe_psql('postgres',
> "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr
> application_name=$appname' PUBLICATION tap_pub WITH (streaming = on,
> min_apply_delay = '3s')"
>
>
> I think there should be a comment here highlighting that you are setting up a
> subscriber time delay of 3 seconds, and then later you can better describe the
> parameters for the checking functions...
Added a comment for CREATE SUBSCRIPTION command.
> e.g. (add this comment)
> # verifies that the subscriber lags the publisher by at least 3 seconds
> check_apply_delay_time($node_publisher, $node_subscriber, '5', '3');
>
> e.g.
> # verifies that the subscriber lags the publisher by at least 3 seconds
> check_apply_delay_time($node_publisher, $node_subscriber, '8', '3');
Added.
> ~~~
>
> 12.
> # Test whether ALTER SUBSCRIPTION changes the delayed time of the apply
> worker # (1 day 1 minute).
> $node_subscriber->safe_psql('postgres',
> "ALTER SUBSCRIPTION tap_sub SET (min_apply_delay = 86460000)"
> );
>
> Update the comment with another note.
> # Note - The extra 1 min is to account for any decoding/network overhead.
Okay, added the comment. In general, TAP tests
fail if we wait for more than 3 minutes. Then,
we should think setting the maximum consumed time
more than 3 minutes is safe. For example, if
(which should not happen usually, but)
we consumed more than 1 minutes between this ALTER SUBSCRIPTION SET
and below check_apply_delay_log() then, the test will fail.
So made the extra time bigger.
> ~~~
>
> 13.
> # Make sure we have long enough min_apply_delay after the ALTER command
> check_apply_delay_log($node_subscriber, "logical replication apply delay",
> "80000000");
>
> IMO the expectation of 1 day (86460000 ms) wait time might be a better number
> for your "expected" value.
>
> So update the comment/call like this:
>
> # Make sure the apply worker knows to wait for more than 1 day (86400000 ms)
> check_apply_delay_log($node_subscriber, "logical replication apply delay",
> "86400000");
Updated the comment and the function call.
Kindly have a look at the updated patch v17.
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v17-0001-Time-delayed-logical-replication-subscriber.patch (80.6K, ../../TYCPR01MB8373F5162C7A0E6224670CF0EDC49@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v17-0001-Time-delayed-logical-replication-subscriber.patch)
download | inline diff:
From 71c11435538e5cf6fab7e507f85d91de602907a1 Mon Sep 17 00:00:00 2001
From: Takamichi Osumi <[email protected]>
Date: Thu, 19 Jan 2023 06:09:50 +0000
Subject: [PATCH v17] Time-delayed logical replication subscriber
Similar to physical replication, a time-delayed copy of the data for
logical replication is useful for some scenarios (particularly to fix
errors that might cause data loss).
This patch implements a new subscription parameter called 'min_apply_delay'.
If the subscription sets min_apply_delay parameter, the logical
replication worker will delay the transaction commit for min_apply_delay
milliseconds.
The delay is calculated between the WAL time stamp and the current time
on the subscriber.
The delay occurs only on WAL records for transaction begins. The main
reason is to avoid keeping a transaction open for a long time. Regular
and prepared transactions are covered. Streamed transactions are also
covered.
Prohibit the combination of this feature and parallel streaming mode.
Author: Euler Taveira
Discussion: https://postgr.es/m/CAB-JLwYOYwL=XTyAXKiH5CtM_Vm8KjKh7aaitCKvmCh4rzr5pQ@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/config.sgml | 13 ++
doc/src/sgml/logical-replication.sgml | 7 +
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 56 ++++-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 7 +-
src/backend/commands/subscriptioncmds.c | 92 ++++++++-
.../replication/logical/applyparallelworker.c | 3 +-
src/backend/replication/logical/worker.c | 161 +++++++++++++--
src/backend/utils/adt/timestamp.c | 29 +++
src/bin/pg_dump/pg_dump.c | 15 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 9 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 3 +
src/include/datatype/timestamp.h | 2 +
src/include/replication/worker_internal.h | 2 +-
src/include/utils/timestamp.h | 2 +
src/test/regress/expected/subscription.out | 181 +++++++++-------
src/test/regress/sql/subscription.sql | 24 +++
src/test/subscription/meson.build | 1 +
src/test/subscription/t/032_apply_delay.pl | 195 ++++++++++++++++++
23 files changed, 720 insertions(+), 102 deletions(-)
create mode 100644 src/test/subscription/t/032_apply_delay.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index c1e4048054..bf3c05241c 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7873,6 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subminapplydelay</structfield> <type>int8</type>
+ </para>
+ <para>
+ The length of time (ms) to delay the application of changes.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subname</structfield> <type>name</type>
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 89d53f2a64..13dd422c25 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4753,6 +4753,19 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
the <filename>postgresql.conf</filename> file or on the server
command line.
</para>
+ <para>
+ For time-delayed logical replication (i.e. when the subscription is
+ created with parameter min_apply_delay > 0), the apply worker sends a
+ Standby Status Update message to the publisher with a period of
+ <literal>wal_receiver_status_interval</literal>. Make sure to set
+ <literal>wal_receiver_status_interval</literal> less than the
+ <literal>wal_sender_timeout</literal> on the publisher, otherwise, the
+ walsender will repeatedly terminate due to the timeout errors. If
+ <literal>wal_receiver_status_interval</literal> is set to zero, the apply
+ worker doesn't send any feedback messages during the subscriber's
+ <literal>min_apply_delay</literal> period. See
+ <xref linkend="sql-createsubscription"/> for details.
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index f4b4e641be..9bfbb3b61d 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -247,6 +247,13 @@
target table.
</para>
+ <para>
+ The subscriber replication can be instructed to lag behind the publisher
+ side changes by specifying the <literal>min_apply_delay</literal>
+ subscription parameter. See <xref linkend="sql-createsubscription"/> for
+ details.
+ </para>
+
<sect2 id="logical-replication-subscription-slot">
<title>Replication Slot Management</title>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index ad93553a1d..1c6e9dd2d1 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -213,8 +213,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
<literal>binary</literal>, <literal>streaming</literal>,
- <literal>disable_on_error</literal>, and
- <literal>origin</literal>.
+ <literal>disable_on_error</literal>,
+ <literal>origin</literal>, and
+ <literal>min_apply_delay</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index eba72c6af6..ac0d477974 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -349,7 +349,44 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
- </variablelist></para>
+
+ <varlistentry>
+ <term><literal>min_apply_delay</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ By default, the subscriber applies changes as soon as possible. This
+ parameter allows the user to delay the application of changes by a
+ specified amount of time. If the value is specified without units, it
+ is taken as milliseconds. The default is zero (no delay).
+ </para>
+ <para>
+ Any delay occurs only on WAL records for transaction begins after all
+ initial table synchronization has finished. The delay is calculated
+ between the WAL timestamp as written on the publisher and the current
+ time on the subscriber. Any overhead of time spent in logical decoding
+ and in transferring the transaction may reduce the actual wait time.
+ It is also possible that the overhead already execeeds the requested
+ <literal>min_apply_delay</literal> value, in which case no additional
+ wait is necessary. If the system clocks on publisher and subscriber
+ are not synchronized, this may lead to apply changes earlier than
+ expected, but this is not a major issue because this parameter is
+ typically much larger than the time deviations between servers. Note
+ that if this parameter is set to a long delay, the replication will
+ stop if the replication slot falls behind the current LSN by more than
+ <link linkend="guc-max-slot-wal-keep-size"><literal>max_slot_wal_keep_size</literal></link>.
+ </para>
+ <warning>
+ <para>
+ Delaying the replication can mean there is a much longer time between making
+ a change on the publisher, and that change being committed on the subscriber.
+ This can have a big impact on synchronous replication.
+ See <xref linkend="guc-synchronous-commit"/>.
+ </para>
+ </warning>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
@@ -413,6 +450,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
published with different column lists are not supported.
</para>
+ <para>
+ A non-zero <literal>min_apply_delay</literal> parameter is not allowed when streaming
+ in parallel mode.
+ </para>
+
<para>
We allow non-existent publications to be specified so that users can add
those later. This means
@@ -472,6 +514,18 @@ CREATE SUBSCRIPTION mysub
PUBLICATION insert_only
WITH (enabled = false);
</programlisting></para>
+
+ <para>
+ Create a subscription to a remote server that replicates tables in
+ the <literal>mypub</literal> publication and starts replicating immediately
+ on commit. Pre-existing data is not copied. The application of changes is
+ delayed by 4 hours.
+<programlisting>
+CREATE SUBSCRIPTION mysub
+ CONNECTION 'host=192.0.2.4 port=5432 user=foo dbname=foodb'
+ PUBLICATION mypub
+ WITH (copy_data = false, min_apply_delay = '4h');
+</programlisting></para>
</refsect1>
<refsect1>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index a56ae311c3..c767cc1c3a 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -64,6 +64,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->oid = subid;
sub->dbid = subform->subdbid;
sub->skiplsn = subform->subskiplsn;
+ sub->minapplydelay = subform->subminapplydelay;
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 8608e3fa5b..317c2010cb 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1299,9 +1299,10 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
- subbinary, substream, subtwophasestate, subdisableonerr,
- subslotname, subsynccommit, subpublications, suborigin)
+GRANT SELECT (oid, subdbid, subskiplsn, subminapplydelay, subname, subowner,
+ subenabled, subbinary, substream, subtwophasestate,
+ subdisableonerr, subslotname, subsynccommit, subpublications,
+ suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index baff00dd74..118842f9ff 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -66,6 +66,7 @@
#define SUBOPT_DISABLE_ON_ERR 0x00000400
#define SUBOPT_LSN 0x00000800
#define SUBOPT_ORIGIN 0x00001000
+#define SUBOPT_MIN_APPLY_DELAY 0x00002000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -90,6 +91,7 @@ typedef struct SubOpts
bool disableonerr;
char *origin;
XLogRecPtr lsn;
+ int min_apply_delay;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -146,6 +148,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->disableonerr = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY))
+ opts->min_apply_delay = 0;
/* Parse options */
foreach(lc, stmt_options)
@@ -324,6 +328,43 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ strcmp(defel->defname, "min_apply_delay") == 0)
+ {
+ char *val,
+ *tmp;
+ Interval *interval;
+ int64 ms;
+
+ if (IsSet(opts->specified_opts, SUBOPT_MIN_APPLY_DELAY))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_MIN_APPLY_DELAY;
+ tmp = defGetString(defel);
+
+ /*
+ * If no unit was specified, then explicitly add 'ms' otherwise
+ * the interval_in function would assume 'seconds'.
+ */
+ if (strspn(tmp, "-0123456789 ") == strlen(tmp))
+ val = psprintf("%sms", tmp);
+ else
+ val = tmp;
+
+ interval = DatumGetIntervalP(DirectFunctionCall3(interval_in,
+ CStringGetDatum(val),
+ ObjectIdGetDatum(InvalidOid),
+ Int32GetDatum(-1)));
+
+ ms = interval2ms(interval);
+ if (ms < 0 || ms > INT_MAX)
+ ereport(ERROR,
+ errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg("%lld ms is outside the valid range for parameter \"%s\" (0 .. %d)",
+ (long long) ms, "min_apply_delay", INT_MAX));
+
+ opts->min_apply_delay = ms;
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -404,6 +445,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * The combination of parallel streaming mode and min_apply_delay is not
+ * allowed.
+ */
+ if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ opts->min_apply_delay > 0)
+ {
+ if (opts->streaming == LOGICALREP_STREAM_PARALLEL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s and %s are mutually exclusive options",
+ "min_apply_delay > 0", "streaming = parallel"));
+ }
}
/*
@@ -560,7 +615,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
- SUBOPT_DISABLE_ON_ERR | SUBOPT_ORIGIN);
+ SUBOPT_DISABLE_ON_ERR | SUBOPT_ORIGIN |
+ SUBOPT_MIN_APPLY_DELAY);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -625,6 +681,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_oid - 1] = ObjectIdGetDatum(subid);
values[Anum_pg_subscription_subdbid - 1] = ObjectIdGetDatum(MyDatabaseId);
values[Anum_pg_subscription_subskiplsn - 1] = LSNGetDatum(InvalidXLogRecPtr);
+ values[Anum_pg_subscription_subminapplydelay - 1] = Int64GetDatum(opts.min_apply_delay);
values[Anum_pg_subscription_subname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
@@ -1054,7 +1111,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
- SUBOPT_ORIGIN);
+ SUBOPT_ORIGIN | SUBOPT_MIN_APPLY_DELAY);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1098,6 +1155,18 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
if (IsSet(opts.specified_opts, SUBOPT_STREAMING))
{
+ /*
+ * The combination of parallel streaming mode and
+ * min_apply_delay is not allowed.
+ */
+ if (opts.streaming == LOGICALREP_STREAM_PARALLEL)
+ if ((IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) && opts.min_apply_delay > 0) ||
+ (!IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) && sub->minapplydelay > 0))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s mode for subscription with %s",
+ "streaming = parallel", "min_apply_delay"));
+
values[Anum_pg_subscription_substream - 1] =
CharGetDatum(opts.streaming);
replaces[Anum_pg_subscription_substream - 1] = true;
@@ -1111,6 +1180,25 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
= true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY))
+ {
+ /*
+ * The combination of parallel streaming mode and
+ * min_apply_delay is not allowed.
+ */
+ if (opts.min_apply_delay > 0)
+ if ((IsSet(opts.specified_opts, SUBOPT_STREAMING) && opts.streaming == LOGICALREP_STREAM_PARALLEL) ||
+ (!IsSet(opts.specified_opts, SUBOPT_STREAMING) && sub->stream == LOGICALREP_STREAM_PARALLEL))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s for subscription in %s mode",
+ "min_apply_delay", "streaming = parallel"));
+
+ values[Anum_pg_subscription_subminapplydelay - 1] =
+ Int64GetDatum(opts.min_apply_delay);
+ replaces[Anum_pg_subscription_subminapplydelay - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index 3579e704fe..7302bce7a0 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -704,7 +704,8 @@ pa_process_spooled_messages_if_required(void)
{
apply_spooled_messages(&MyParallelShared->fileset,
MyParallelShared->xid,
- InvalidXLogRecPtr);
+ InvalidXLogRecPtr,
+ 0);
pa_set_fileset_state(MyParallelShared, FS_EMPTY);
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a0084c7ef6..7512ae5b7d 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -318,6 +318,17 @@ static List *on_commit_wakeup_workers_subids = NIL;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/*
+ * In order to avoid walsender timeout for time-delayed replication the worker
+ * process keeps sending feedback messages during the delay period.
+ * Meanwhile, the feature delays the apply before starting the
+ * transaction and thus we don't write WALs for the suspended changes during
+ * the wait. When the worker process sends a feedback message
+ * during the delay, we should not make positions of the flushed and apply LSN
+ * overwritten by the last received latest LSN. See send_feedback() for details.
+ */
+static XLogRecPtr last_received = InvalidXLogRecPtr;
+
/* fields valid only when processing streamed transaction */
static bool in_streamed_transaction = false;
@@ -388,10 +399,13 @@ static void stream_write_change(char action, StringInfo s);
static void stream_open_and_write_change(TransactionId xid, char action, StringInfo s);
static void stream_close_file(void);
-static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
+static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply,
+ bool in_delayed_apply);
static void DisableSubscriptionAndExit(void);
+static void maybe_delay_apply(TimestampTz finish_ts);
+
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
@@ -998,6 +1012,105 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
ExecStoreVirtualTuple(slot);
}
+/*
+ * When min_apply_delay parameter is set on the subscriber, we wait long enough
+ * to make sure a transaction is applied at least that interval behind the
+ * publisher.
+ *
+ * While the physical replication applies the delay at commit time, this
+ * feature applies the delay for the next transaction but before starting the
+ * transaction. This is mainly because keeping a transaction that conducted
+ * write operations open for a long time results in some issues such as bloat
+ * and locks.
+ *
+ * The min_apply_delay parameter will take effect only after all tables are in
+ * READY state.
+ *
+ * finish_ts is the commit/prepare time of both regular (non-streamed) and
+ * streamed transactions. Unlike the regular (non-streamed) cases, the delay
+ * is applied in a STREAM COMMIT/STREAM PREPARE message for streamed
+ * transactions. The STREAM START message does not contain a commit/prepare
+ * time (it will be available when the in-progress transaction finishes).
+ * Hence, it's not appropriate to apply a delay at the time.
+ */
+static void
+maybe_delay_apply(TimestampTz finish_ts)
+{
+ Assert(finish_ts > 0);
+
+ /* Nothing to do if no delay set */
+ if (!MySubscription->minapplydelay)
+ return;
+
+ /*
+ * The min_apply_delay parameter is ignored until all tablesync workers
+ * have reached READY state. This is because if we allowed the delay
+ * during the catchup phase, then once we reached the limit of tablesync
+ * workers it would impose a delay for each subsequent worker. That would
+ * cause initial table synchronization completion to take a long time.
+ */
+ if (!AllTablesyncsReady())
+ return;
+
+ /* Apply the delay by the latch mechanism */
+ while (true)
+ {
+ long diffms;
+
+ ResetLatch(MyLatch);
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* This might change wal_receiver_status_interval */
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /*
+ * Before calculating the time duration, reload the catalog if needed.
+ */
+ if (!in_remote_transaction && !in_streamed_transaction)
+ {
+ AcceptInvalidationMessages();
+ maybe_reread_subscription();
+ }
+
+ diffms = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ TimestampTzPlusMilliseconds(finish_ts, MySubscription->minapplydelay));
+
+ /*
+ * Exit without arming the latch if it's already past time to apply
+ * this transaction.
+ */
+ if (diffms <= 0)
+ break;
+
+ elog(DEBUG2, "logical replication apply delay: %ld ms", diffms);
+
+ /*
+ * Call send_feedback() to prevent the publisher from exiting by
+ * timeout during the delay, when wal_receiver_status_interval is
+ * available.
+ */
+ if (wal_receiver_status_interval > 0
+ && diffms > wal_receiver_status_interval * 1000)
+ {
+ WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ (long) wal_receiver_status_interval * 1000,
+ WAIT_EVENT_RECOVERY_APPLY_DELAY);
+ send_feedback(last_received, true, false, true);
+ }
+ else
+ WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ diffms,
+ WAIT_EVENT_RECOVERY_APPLY_DELAY);
+ }
+}
+
/*
* Handle BEGIN message.
*/
@@ -1012,6 +1125,9 @@ apply_handle_begin(StringInfo s)
logicalrep_read_begin(s, &begin_data);
set_apply_error_context_xact(begin_data.xid, begin_data.final_lsn);
+ /* Should we delay the current transaction? */
+ maybe_delay_apply(begin_data.committime);
+
remote_final_lsn = begin_data.final_lsn;
maybe_start_skipping_changes(begin_data.final_lsn);
@@ -1069,6 +1185,9 @@ apply_handle_begin_prepare(StringInfo s)
logicalrep_read_begin_prepare(s, &begin_data);
set_apply_error_context_xact(begin_data.xid, begin_data.prepare_lsn);
+ /* Should we delay the current prepared transaction? */
+ maybe_delay_apply(begin_data.prepare_time);
+
remote_final_lsn = begin_data.prepare_lsn;
maybe_start_skipping_changes(begin_data.prepare_lsn);
@@ -1316,7 +1435,8 @@ apply_handle_stream_prepare(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset,
- prepare_data.xid, prepare_data.prepare_lsn);
+ prepare_data.xid, prepare_data.prepare_lsn,
+ prepare_data.prepare_time);
/* Mark the transaction as prepared. */
apply_handle_prepare_internal(&prepare_data);
@@ -2010,10 +2130,13 @@ ensure_last_message(FileSet *stream_fileset, TransactionId xid, int fileno,
/*
* Common spoolfile processing.
+ *
+ * The commit/prepare time for streaming transaction is required to achieve
+ * time-delayed replication.
*/
void
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn, TimestampTz finish_ts)
{
StringInfoData s2;
int nchanges;
@@ -2024,6 +2147,10 @@ apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
int fileno;
off_t offset;
+ /* Should we delay the current transaction? */
+ if (finish_ts)
+ maybe_delay_apply(finish_ts);
+
if (!am_parallel_apply_worker())
maybe_start_skipping_changes(lsn);
@@ -2173,7 +2300,7 @@ apply_handle_stream_commit(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset, xid,
- commit_data.commit_lsn);
+ commit_data.commit_lsn, commit_data.committime);
apply_handle_commit_internal(&commit_data);
@@ -3446,7 +3573,7 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
* Apply main loop.
*/
static void
-LogicalRepApplyLoop(XLogRecPtr last_received)
+LogicalRepApplyLoop(void)
{
TimestampTz last_recv_timestamp = GetCurrentTimestamp();
bool ping_sent = false;
@@ -3567,7 +3694,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
if (last_received < end_lsn)
last_received = end_lsn;
- send_feedback(last_received, reply_requested, false);
+ send_feedback(last_received, reply_requested, false, false);
UpdateWorkerStats(last_received, timestamp, true);
}
/* other message types are purposefully ignored */
@@ -3580,7 +3707,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/* confirm all writes so far */
- send_feedback(last_received, false, false);
+ send_feedback(last_received, false, false, false);
if (!in_remote_transaction && !in_streamed_transaction)
{
@@ -3677,7 +3804,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
}
- send_feedback(last_received, requestReply, requestReply);
+ send_feedback(last_received, requestReply, requestReply, false);
/*
* Force reporting to ensure long idle periods don't lead to
@@ -3707,7 +3834,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
* to send a response to avoid timeouts.
*/
static void
-send_feedback(XLogRecPtr recvpos, bool force, bool requestReply)
+send_feedback(XLogRecPtr recvpos, bool force, bool requestReply, bool in_delayed_apply)
{
static StringInfo reply_message = NULL;
static TimestampTz send_time = 0;
@@ -3737,8 +3864,15 @@ send_feedback(XLogRecPtr recvpos, bool force, bool requestReply)
/*
* No outstanding transactions to flush, we can report the latest received
* position. This is important for synchronous replication.
+ *
+ * If the subscriber side apply is delayed (because of time-delayed
+ * replication) then do not tell the publisher that the received latest
+ * LSN is already applied and flushed, otherwise, it leads to the
+ * publisher side making a wrong assumption of logical replication
+ * progress. Instead, we just send a feedback message to avoid a publisher
+ * timeout during the delay.
*/
- if (!have_pending_txes)
+ if (!have_pending_txes && !in_delayed_apply)
flushpos = writepos = recvpos;
if (writepos < last_writepos)
@@ -4354,11 +4488,11 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname)
* of system resource error and are not repeatable.
*/
static void
-start_apply(XLogRecPtr origin_startpos)
+start_apply(void)
{
PG_TRY();
{
- LogicalRepApplyLoop(origin_startpos);
+ LogicalRepApplyLoop();
}
PG_CATCH();
{
@@ -4645,7 +4779,8 @@ ApplyWorkerMain(Datum main_arg)
}
/* Run the main loop. */
- start_apply(origin_startpos);
+ last_received = origin_startpos;
+ start_apply();
proc_exit(0);
}
diff --git a/src/backend/utils/adt/timestamp.c b/src/backend/utils/adt/timestamp.c
index 928c330897..422e6ad0fa 100644
--- a/src/backend/utils/adt/timestamp.c
+++ b/src/backend/utils/adt/timestamp.c
@@ -2431,6 +2431,35 @@ interval_cmp_internal(const Interval *interval1, const Interval *interval2)
return int128_compare(span1, span2);
}
+/*
+ * Returns the number of milliseconds in the specified Interval.
+ */
+int64
+interval2ms(const Interval *interval)
+{
+ int64 days;
+ int64 ms;
+ int64 result;
+
+ days = interval->month * INT64CONST(30);
+ days += interval->day;
+
+ /* Detect whether the value of interval can cause an overflow */
+ if (pg_mul_s64_overflow(days, MSECS_PER_DAY, &result))
+ ereport(ERROR,
+ errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg("bigint out of range"));
+
+ /* Adds portion time (in ms) to the previous result */
+ ms = interval->time / INT64CONST(1000);
+ if (pg_add_s64_overflow(result, ms, &result))
+ ereport(ERROR,
+ errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg("bigint out of range"));
+
+ return result;
+}
+
Datum
interval_eq(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 527c7651ab..c0f69cb43b 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4494,6 +4494,7 @@ getSubscriptions(Archive *fout)
int i_subsynccommit;
int i_subpublications;
int i_subbinary;
+ int i_subminapplydelay;
int i,
ntups;
@@ -4546,9 +4547,13 @@ getSubscriptions(Archive *fout)
LOGICALREP_TWOPHASE_STATE_DISABLED);
if (fout->remoteVersion >= 160000)
- appendPQExpBufferStr(query, " s.suborigin\n");
+ appendPQExpBufferStr(query,
+ " s.suborigin,\n"
+ " s.subminapplydelay\n");
else
- appendPQExpBuffer(query, " '%s' AS suborigin\n", LOGICALREP_ORIGIN_ANY);
+ appendPQExpBuffer(query, " '%s' AS suborigin,\n"
+ " 0 AS subminapplydelay\n",
+ LOGICALREP_ORIGIN_ANY);
appendPQExpBufferStr(query,
"FROM pg_subscription s\n"
@@ -4576,6 +4581,7 @@ getSubscriptions(Archive *fout)
i_subtwophasestate = PQfnumber(res, "subtwophasestate");
i_subdisableonerr = PQfnumber(res, "subdisableonerr");
i_suborigin = PQfnumber(res, "suborigin");
+ i_subminapplydelay = PQfnumber(res, "subminapplydelay");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4606,6 +4612,8 @@ getSubscriptions(Archive *fout)
subinfo[i].subdisableonerr =
pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
subinfo[i].suborigin = pg_strdup(PQgetvalue(res, i, i_suborigin));
+ subinfo[i].subminapplydelay =
+ strtoi64(PQgetvalue(res, i, i_subminapplydelay), NULL, 10);
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -4687,6 +4695,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
+ if (subinfo->subminapplydelay > 0)
+ appendPQExpBuffer(query, ", min_apply_delay = '" INT64_FORMAT " ms'", subinfo->subminapplydelay);
+
appendPQExpBufferStr(query, ");\n");
if (subinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index e7cbd8d7ed..e2525f70ab 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -661,6 +661,7 @@ typedef struct _SubscriptionInfo
char *subdisableonerr;
char *suborigin;
char *subsynccommit;
+ int64 subminapplydelay;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index c8a0bb7b3a..8a27063bed 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6472,7 +6472,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false, false, false, false};
+ false, false, false, false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6527,10 +6527,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Two-phase commit"),
gettext_noop("Disable on error"));
+ /* Origin and min_apply_delay are only supported in v16 and higher */
if (pset.sversion >= 160000)
appendPQExpBuffer(&buf,
- ", suborigin AS \"%s\"\n",
- gettext_noop("Origin"));
+ ", suborigin AS \"%s\"\n"
+ ", subminapplydelay AS \"%s\"\n",
+ gettext_noop("Origin"),
+ gettext_noop("Min apply delay (ms)"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 5e1882eaea..e8b9a43a47 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1925,7 +1925,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "origin", "slot_name",
+ COMPLETE_WITH("binary", "disable_on_error", "min_apply_delay", "origin", "slot_name",
"streaming", "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
@@ -3268,7 +3268,7 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "origin", "slot_name",
+ "disable_on_error", "enabled", "min_apply_delay", "origin", "slot_name",
"streaming", "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index b0f2a1705d..e06f35c037 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -70,6 +70,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
XLogRecPtr subskiplsn; /* All changes finished at this LSN are
* skipped */
+ int64 subminapplydelay; /* Replication apply delay (ms) */
+
NameData subname; /* Name of the subscription */
Oid subowner BKI_LOOKUP(pg_authid); /* Owner of the subscription */
@@ -120,6 +122,7 @@ typedef struct Subscription
* in */
XLogRecPtr skiplsn; /* All changes finished at this LSN are
* skipped */
+ int64 minapplydelay; /* Replication apply delay (ms) */
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
diff --git a/src/include/datatype/timestamp.h b/src/include/datatype/timestamp.h
index 21a37e21e9..8b368af299 100644
--- a/src/include/datatype/timestamp.h
+++ b/src/include/datatype/timestamp.h
@@ -127,6 +127,8 @@ struct pg_itm_in
#define SECS_PER_MINUTE 60
#define MINS_PER_HOUR 60
+#define MSECS_PER_DAY INT64CONST(86400000)
+
#define USECS_PER_DAY INT64CONST(86400000000)
#define USECS_PER_HOUR INT64CONST(3600000000)
#define USECS_PER_MINUTE INT64CONST(60000000)
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index dc87a4edd1..3dc09d1a4c 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -255,7 +255,7 @@ extern void stream_stop_internal(TransactionId xid);
/* Common streaming function to apply all the spooled messages */
extern void apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn);
+ XLogRecPtr lsn, TimestampTz finish_ts);
extern void apply_dispatch(StringInfo s);
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index 42f802bb9d..534051fe13 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -102,6 +102,8 @@ extern bool TimestampDifferenceExceeds(TimestampTz start_time,
TimestampTz stop_time,
int msec);
+extern int64 interval2ms(const Interval *interval);
+
extern TimestampTz time_t_to_timestamptz(pg_time_t tm);
extern pg_time_t timestamptz_to_time_t(TimestampTz t);
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 4e5cb0d3a9..eb25c286a2 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -114,18 +114,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -135,10 +135,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -155,10 +155,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -167,10 +167,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -202,10 +202,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -239,19 +239,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -263,27 +263,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -298,10 +298,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -316,10 +316,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -355,10 +355,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -367,10 +367,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -380,10 +380,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,20 +396,57 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail -- min_apply_delay must be a non-negative integer
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = foo);
+ERROR: invalid input syntax for type interval: "foo"
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = -1);
+ERROR: -1 ms is outside the valid range for parameter "min_apply_delay" (0 .. 2147483647)
+-- fail - utilizing streaming = parallel with time-delayed replication is not supported.
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = parallel, min_apply_delay = '1 day');
+ERROR: min_apply_delay > 0 and streaming = parallel are mutually exclusive options
+-- success -- min_apply_delay value without unit is taken as milliseconds
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = 123);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 123 | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+-- success -- min_apply_delay interval is converted into ms and stored as an integer
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = '4h 27min 35s');
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 16055000 | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+-- fail - alter subscription with streaming = parallel should fail when time-delayed replication is set.
+ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
+ERROR: cannot enable streaming = parallel mode for subscription with min_apply_delay
+-- fail - alter subscription with min_apply_delay should fail when streaming = parallel is set.
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 0, streaming = parallel);
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 123);
+ERROR: cannot enable min_apply_delay for subscription in streaming = parallel mode
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
RESET SESSION AUTHORIZATION;
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 5f27b7d776..d4c2a1987e 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -279,6 +279,30 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail -- min_apply_delay must be a non-negative integer
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = foo);
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = -1);
+
+-- fail - utilizing streaming = parallel with time-delayed replication is not supported.
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = parallel, min_apply_delay = '1 day');
+
+-- success -- min_apply_delay value without unit is taken as milliseconds
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = 123);
+\dRs+
+
+-- success -- min_apply_delay interval is converted into ms and stored as an integer
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = '4h 27min 35s');
+\dRs+
+
+-- fail - alter subscription with streaming = parallel should fail when time-delayed replication is set.
+ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
+
+-- fail - alter subscription with min_apply_delay should fail when streaming = parallel is set.
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 0, streaming = parallel);
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 123);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index 3db0fdfd96..a186876eb4 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -38,6 +38,7 @@ tests += {
't/029_on_error.pl',
't/030_origin.pl',
't/031_column_list.pl',
+ 't/032_apply_delay.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/032_apply_delay.pl b/src/test/subscription/t/032_apply_delay.pl
new file mode 100644
index 0000000000..7a84a3f0e2
--- /dev/null
+++ b/src/test/subscription/t/032_apply_delay.pl
@@ -0,0 +1,195 @@
+
+# Copyright (c) 2023, PostgreSQL Global Development Group
+
+# Test replication apply delay
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $log_location = 0;
+
+# Confirm the time-delayed replication has been effective from the server log
+# message where the apply worker emits for applying delay. When necessary,
+# verifies that the current worker's delayed time is sufficiently bigger than
+# the expected value, in order to check any update of the min_apply_delay.
+sub check_apply_delay_log
+{
+ my ($node_subscriber, $expected) = @_;
+ $expected = 0 unless defined $expected;
+
+ my $old_log_location = $log_location;
+
+ $log_location = $node_subscriber->wait_for_log(qr/logical replication apply delay/, $log_location);
+
+ cmp_ok($log_location, '>', $old_log_location,
+ "logfile contains triggered logical replication apply delay"
+ );
+
+ if ($expected > 0)
+ {
+ # Get the delay time from the server log
+ my $contents = slurp_file($node_subscriber->logfile, $old_log_location);
+ $contents =~
+ qr/logical replication apply delay: (\d+) ms/
+ or die "could not get the apply worker wait time";
+ my $logged_delay = $1;
+
+ # Is it larger than expected?
+ cmp_ok($logged_delay, '>', $expected,
+ "The apply worker wait time has expected duration"
+ );
+ }
+}
+
+# Compare inserted time on the publisher with applied time on the subscriber to
+# confirm the latter is applied after expected time.
+sub check_apply_delay_time
+{
+ my ($node_publisher, $node_subscriber, $primary_key, $expected_diffs) = @_;
+
+ my $inserted_time_on_pub = $node_publisher->safe_psql('postgres', qq[
+ SELECT extract(epoch from c) FROM test_tab WHERE a = $primary_key;
+ ]);
+
+ my $inserted_time_on_sub = $node_subscriber->safe_psql('postgres', qq[
+ SELECT extract(epoch from c) FROM test_tab WHERE a = $primary_key;
+ ]);
+
+ cmp_ok($inserted_time_on_sub - $inserted_time_on_pub, '>', $expected_diffs,
+ "The tuple on the subscriber was modified later than the publisher");
+}
+
+# Create publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = debug2");
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar, c timestamptz DEFAULT now())");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)"
+);
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+# The column c must not be published because we want to compare the time
+# difference.
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE test_tab (a, b)");
+
+my $appname = 'tap_sub';
+
+# Create a subscription that applies the trasaction after 3 seconds delay
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on, min_apply_delay = '3s')"
+);
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Check log starting now for logical replication apply delay
+$log_location = -s $node_subscriber->logfile;
+
+my $result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM test_tab");
+is($result, qq(2|1|2), 'check initial data was copied to subscriber');
+
+# New row to trigger apply delay
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (3, 'baz')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (4, 'abc')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (5, 'def')");
+
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM test_tab");
+is($result, qq(5|1|5), 'check if the new rows were applied to subscriber');
+
+# Verify that the apply worker emits the apply delay log
+check_apply_delay_log($node_subscriber);
+
+# Verify that the subscriber lags the publisher by at least 3 seconds
+check_apply_delay_time($node_publisher, $node_subscriber, '5', '3');
+
+# Reduce the amounts of writes for spooling file
+$node_publisher->append_conf('postgres.conf',
+ 'logical_decoding_mode = immediate');
+$node_publisher->reload;
+
+# Run a query to make sure that the reload has taken effect.
+$node_publisher->safe_psql('postgres', q{SELECT 1});
+
+# Test streamed transaction by insert, update and delete
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6, 8) s(i);");
+
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM test_tab");
+is($result, qq(8|1|8), 'check if the new rows were applied to subscriber');
+
+# Verify that the apply worker emits the apply delay log
+check_apply_delay_log($node_subscriber);
+
+# Verify that the subscriber lags the publisher by at least 3 seconds
+check_apply_delay_time($node_publisher, $node_subscriber, '8', '3');
+
+# Test whether ALTER SUBSCRIPTION changes the delayed time of the apply worker
+# (1 day 5 minutes). Note that the extra 5 minute is to account for any
+# decoding/network overhead.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (min_apply_delay = 86700000)"
+);
+
+# New row to trigger apply delay
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (0, 'foobar')");
+
+# Make sure the apply worker knows to wait for more than 1 day
+check_apply_delay_log($node_subscriber, "86400000");
+
+# Disable subscription and the worker should die immediately
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub DISABLE;"
+);
+
+# Wait until worker dies
+my $sub_query =
+ "SELECT count(1) = 0 FROM pg_stat_subscription WHERE subname = 'tap_sub' AND pid IS NOT NULL;";
+$node_subscriber->poll_query_until('postgres', $sub_query)
+ or die "Timed out while waiting for subscriber to die";
+
+# Confirm the suspended record doesn't get applied expectedly by the ALTER
+# DISABLE command.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(a) FROM test_tab WHERE a = 0;");
+is($result, qq(0), "check if the delayed transaction doesn't get applied expectedly");
+
+$node_subscriber->stop;
+$node_publisher->stop;
+
+done_testing();
--
2.30.0
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-19 07:12 Takamichi Osumi (Fujitsu) <[email protected]>
parent: Peter Smith <[email protected]>
2 siblings, 1 reply; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-19 07:12 UTC (permalink / raw)
To: 'Peter Smith' <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Wednesday, January 18, 2023 4:06 PM Peter Smith <[email protected]> wrote:
> Here are my review comments for the latest patch v16-0001. (excluding the
> test code)
Hi, thank you for your review !
> ======
>
> General
>
> 1.
>
> Since the value of min_apply_delay cannot be < 0, I was thinking probably it
> should have been declared everywhere in this patch as a
> uint64 instead of an int64, right?
No, we won't be able to adopt this idea.
It seems that we are not able to use uint for catalog type.
So, can't applying it to the pg_subscription.h definitions
and then similarly Int64GetDatum to store catalog variables
and the argument variable of Int64GetDatum.
Plus, there is a possibility that type Interval becomes negative value,
then we are not able to change the int64 variable to get
the return value of interval2ms().
> ======
>
> Commit message
>
> 2.
>
> If the subscription sets min_apply_delay parameter, the logical replication
> worker will delay the transaction commit for min_apply_delay milliseconds.
>
> ~
>
> IMO there should be another sentence before this just to say that a new
> parameter is being added:
>
> e.g.
> This patch implements a new subscription parameter called
> 'min_apply_delay'.
Added.
> ======
>
> doc/src/sgml/config.sgml
>
> 3.
>
> + <para>
> + For time-delayed logical replication, the apply worker sends a Standby
> + Status Update message to the corresponding publisher per the
> indicated
> + time of this parameter. Therefore, if this parameter is longer than
> + <literal>wal_sender_timeout</literal> on the publisher, then the
> + walsender doesn't get any update message during the delay and
> repeatedly
> + terminates due to the timeout errors. Hence, make sure this parameter
> is
> + shorter than the <literal>wal_sender_timeout</literal> of the
> publisher.
> + If this parameter is set to zero with time-delayed replication, the
> + apply worker doesn't send any feedback messages during the
> + <literal>min_apply_delay</literal>.
> + </para>
>
>
> This paragraph seemed confusing. I think it needs to be reworded to change all
> of the "this parameter" references because there are at least 3 different
> parameters mentioned in this paragraph. e.g. maybe just change them to
> explicitly name the parameter you are talking about.
>
> I also think it needs to mention the ‘min_apply_delay’ subscription parameter
> up-front and then refer to it appropriately.
>
> The end result might be something like I wrote below (this is just my guess ?
> probably you can word it better).
>
> SUGGESTION
> For time-delayed logical replication (i.e. when the subscription is created with
> parameter min_apply_delay > 0), the apply worker sends a Standby Status
> Update message to the publisher with a period of wal_receiver_status_interval .
> Make sure to set wal_receiver_status_interval less than the
> wal_sender_timeout on the publisher, otherwise, the walsender will repeatedly
> terminate due to the timeout errors. If wal_receiver_status_interval is set to zero,
> the apply worker doesn't send any feedback messages during the subscriber’s
> min_apply_delay period.
Applied. Also, I added one reference for min_apply_delay parameter
at the end of this description.
> ======
>
> doc/src/sgml/ref/create_subscription.sgml
>
> 4.
>
> + <para>
> + By default, the subscriber applies changes as soon as possible. As
> + with the physical replication feature
> + (<xref linkend="guc-recovery-min-apply-delay"/>), it can be
> useful to
> + have a time-delayed logical replica. This parameter lets the user to
> + delay the application of changes by a specified amount of
> time. If this
> + value is specified without units, it is taken as milliseconds. The
> + default is zero(no delay).
> + </para>
>
> 4a.
> As with the physical replication feature (recovery_min_apply_delay), it can be
> useful to have a time-delayed logical replica.
>
> IMO not sure that the above sentence is necessary. It seems only to be saying
> that this parameter can be useful. Why do we need to say that?
Removed the sentence.
> ~
>
> 4b.
> "This parameter lets the user to delay" -> "This parameter lets the user delay"
> OR
> "This parameter lets the user to delay" -> "This parameter allows the user to
> delay"
Fixed.
> ~
>
> 4c.
> "If this value is specified without units" -> "If the value is specified without
> units"
Fixed.
> ~
>
> 4d.
> "zero(no delay)." -> "zero (no delay)."
Fixed.
> ----
>
> 5.
>
> + <para>
> + The delay occurs only on WAL records for transaction begins and
> after
> + the initial table synchronization. It is possible that the
> + replication delay between publisher and subscriber exceeds the
> value
> + of this parameter, in which case no delay is added. Note that the
> + delay is calculated between the WAL time stamp as written on
> + publisher and the current time on the subscriber. Time
> spent in logical
> + decoding and in transferring the transaction may reduce the
> actual wait
> + time. If the system clocks on publisher and subscriber are not
> + synchronized, this may lead to apply changes earlier than
> expected,
> + but this is not a major issue because this parameter is
> typically much
> + larger than the time deviations between servers. Note that if this
> + parameter is set to a long delay, the replication will stop if the
> + replication slot falls behind the current LSN by more than
> + <link
> linkend="guc-max-slot-wal-keep-size"><literal>max_slot_wal_keep_size</
> literal></link>.
> + </para>
>
> I think the first part can be reworded slightly. See what you think about the
> suggestion below.
>
> SUGGESTION
> Any delay occurs only on WAL records for transaction begins after all initial
> table synchronization has finished. The delay is calculated between the WAL
> timestamp as written on the publisher and the current time on the subscriber.
> Any overhead of time spent in logical decoding and in transferring the
> transaction may reduce the actual wait time.
> It is also possible that the overhead already exceeds the requested
> 'min_apply_delay' value, in which case no additional wait is necessary. If the
> system clocks...
Addressed.
> ----
>
> 6.
>
> + <para>
> + Setting streaming to <literal>parallel</literal> mode and
> <literal>min_apply_delay</literal>
> + simultaneously is not supported.
> + </para>
>
> SUGGESTION
> A non-zero min_apply_delay parameter is not allowed when streaming in
> parallel mode.
Applied.
> ======
>
> src/backend/commands/subscriptioncmds.c
>
> 7. parse_subscription_options
>
> @@ -404,6 +445,17 @@ parse_subscription_options(ParseState *pstate, List
> *stmt_options,
> "slot_name = NONE", "create_slot = false")));
> }
> }
> +
> + /* Test the combination of streaming mode and min_apply_delay */ if
> + (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
> + opts->min_apply_delay > 0)
> + {
> + if (opts->streaming == LOGICALREP_STREAM_PARALLEL)
> ereport(ERROR,
> + errcode(ERRCODE_SYNTAX_ERROR), errmsg("%s and %s are mutually
> + exclusive options",
> + "min_apply_delay > 0", "streaming = parallel")); }
>
> SUGGESTION (comment)
> The combination of parallel streaming mode and min_apply_delay is not
> allowed.
Fixed.
> ~~~
>
> 8. AlterSubscription (general)
>
> I observed during testing there are 3 different errors….
>
> At subscription CREATE time you can get this error:
> ERROR: min_apply_delay > 0 and streaming = parallel are mutually exclusive
> options
>
> If you try to ALTER the min_apply_delay when already streaming = parallel you
> can get this error:
> ERROR: cannot enable min_apply_delay for subscription in streaming =
> parallel mode
>
> If you try to ALTER the streaming to be parallel if there is already a
> min_apply_delay > 0 then you can get this error:
> ERROR: cannot enable streaming = parallel mode for subscription with
> min_apply_delay
Yes. This is because the existing error message styles
in AlterSubscription and parse_subscription_options.
The former uses "mutually exclusive" messages consistently,
while the latter does "cannot enable ..." ones.
> ~
>
> IMO there is no need to have 3 different error message texts. I think all these
> cases are explained by just the first text (ERROR:
> min_apply_delay > 0 and streaming = parallel are mutually exclusive
> options)
Then, we followed this kind of formats.
> ~~~
>
> 9. AlterSubscription
>
> @@ -1098,6 +1152,18 @@ AlterSubscription(ParseState *pstate,
> AlterSubscriptionStmt *stmt,
>
> if (IsSet(opts.specified_opts, SUBOPT_STREAMING))
> {
> + /*
> + * Test the combination of streaming mode and
> + * min_apply_delay
> + */
> + if (opts.streaming == LOGICALREP_STREAM_PARALLEL) if
> + ((IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) &&
> opts.min_apply_delay > 0) ||
> + (!IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) &&
> sub->minapplydelay > 0))
> + ereport(ERROR,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("cannot enable %s mode for subscription with %s",
> + "streaming = parallel", "min_apply_delay"));
> +
>
> 9a.
> SUGGESTION (comment)
> The combination of parallel streaming mode and min_apply_delay is not
> allowed.
Fixed.
> ~
>
> 9b.
> (see AlterSubscription general review comment #8 above) Here you can use the
> same comment error message that says min_apply_delay > 0 and streaming =
> parallel are mutually exclusive options.
As described above, we followed the current style in the existing functions.
> ~~~
>
> 10. AlterSubscription
>
> @@ -1111,6 +1177,25 @@ AlterSubscription(ParseState *pstate,
> AlterSubscriptionStmt *stmt,
> = true;
> }
>
> + if (IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY)) {
> + /*
> + * Test the combination of streaming mode and
> + * min_apply_delay
> + */
> + if (opts.min_apply_delay > 0)
> + if ((IsSet(opts.specified_opts, SUBOPT_STREAMING) && opts.streaming
> == LOGICALREP_STREAM_PARALLEL) ||
> + (!IsSet(opts.specified_opts, SUBOPT_STREAMING) && sub->stream ==
> LOGICALREP_STREAM_PARALLEL))
> + ereport(ERROR,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("cannot enable %s for subscription in %s mode",
> + "min_apply_delay", "streaming = parallel"));
> +
> + values[Anum_pg_subscription_subminapplydelay - 1] =
> + Int64GetDatum(opts.min_apply_delay);
> + replaces[Anum_pg_subscription_subminapplydelay - 1] = true; }
>
> 10a.
> SUGGESTION (comment)
> The combination of parallel streaming mode and min_apply_delay is not
> allowed.
Fixed.
> ~
>
> 10b.
> (see AlterSubscription general review comment #8 above) Here you can use the
> same comment error message that says min_apply_delay > 0 and streaming =
> parallel are mutually exclusive options.
Same as 9b.
> ======
>
> .../replication/logical/applyparallelworker.c
>
> 11.
>
> @@ -704,7 +704,8 @@ pa_process_spooled_messages_if_required(void)
> {
> apply_spooled_messages(&MyParallelShared->fileset,
> MyParallelShared->xid,
> - InvalidXLogRecPtr);
> + InvalidXLogRecPtr,
> + 0);
>
> IMO this passing of 0 is a bit strange because it is currently acting like a dummy
> value since the apply_spooled_messages will never make use of the 'finish_ts'
> anyway (since this call is from a parallel apply worker).
>
> I think a better way to code this might be to pass the 0 (same as you are doing
> here) but inside the apply_spooled_messages change the code:
>
> FROM
> if (!am_parallel_apply_worker())
> maybe_delay_apply(finish_ts);
>
> TO
> if (finish_ts)
> maybe_delay_apply(finish_ts);
>
> That does 2 things.
> - It makes the passed-in 0 have some meaning
> - It simplifies the apply_spooled_messages code
Adopted.
> ======
>
> src/backend/replication/logical/worker.c
>
> 12.
>
> @@ -318,6 +318,17 @@ static List *on_commit_wakeup_workers_subids =
> NIL; bool in_remote_transaction = false; static XLogRecPtr
> remote_final_lsn = InvalidXLogRecPtr;
>
> +/*
> + * In order to avoid walsender's timeout during time-delayed
> +replication,
> + * it's necessary to keep sending feedback messages during the delay
> +from the
> + * worker process. Meanwhile, the feature delays the apply before
> +starting the
> + * transaction and thus we don't write WALs for the suspended changes
> +during
> + * the wait. Hence, in the case the worker process sends a feedback
> +message
> + * during the delay, we should not make positions of the flushed and
> +apply LSN
> + * overwritten by the last received latest LSN. See send_feedback()
> for details.
> + */
> +static XLogRecPtr last_received = InvalidXLogRecPtr;
>
> 12a.
> Suggest a small change to the first sentence of the comment.
>
> BEFORE
> In order to avoid walsender's timeout during time-delayed replication, it's
> necessary to keep sending feedback messages during the delay from the
> worker process.
>
> AFTER
> In order to avoid walsender timeout for time-delayed replication the worker
> process keeps sending feedback messages during the delay period.
Fixed.
> ~
>
> 12b.
> "Hence, in the case" -> "When"
Fixed.
> ~~~
>
> 13. forward declare
>
> -static void send_feedback(XLogRecPtr recvpos, bool force, bool
> requestReply);
> +static void send_feedback(XLogRecPtr recvpos, bool force, bool
> requestReply,
> + bool in_delaying_apply);
>
> Change the param name:
>
> "in_delaying_apply" -> "in_delayed_apply” (??)
Changed. The initial intention to append the "in_"
prefix is to make the variable name aligned with
some other variables such as "in_remote_transaction" and
"in_streamed_transaction" that mean the current status
for the transaction. So, until there is a better name proposed,
we can keep it.
> ~~~
>
> 14. maybe_delay_apply
>
> + /* Nothing to do if no delay set */
> + if (MySubscription->minapplydelay <= 0) return;
>
> IIUC min_apply_delay cannot be < 0 so this condition could simply be:
>
> if (!MySubscription->minapplydelay)
> return;
Fixed.
> ~~~
>
> 15. maybe_delay_apply
>
> + /*
> + * The min_apply_delay parameter is ignored until all tablesync workers
> + * have reached READY state. If we allow the delay during the catchup
> + * phase, once we reach the limit of tablesync workers, it will impose
> + a
> + * delay for each subsequent worker. It means it will take a long time
> + to
> + * finish the initial table synchronization.
> + */
> + if (!AllTablesyncsReady())
> + return;
>
> SUGGESTION (slight rewording)
> The min_apply_delay parameter is ignored until all tablesync workers have
> reached READY state. This is because if we allowed the delay during the
> catchup phase, then once we reached the limit of tablesync workers it would
> impose a delay for each subsequent worker. That would cause initial table
> synchronization completion to take a long time.
Fixed.
> ~~~
>
> 16. maybe_delay_apply
>
> + while (true)
> + {
> + long diffms;
> +
> + ResetLatch(MyLatch);
> +
> + CHECK_FOR_INTERRUPTS();
>
> IMO there should be some small explanatory comment here at the top of the
> while loop.
Added.
> ~~~
>
> 17. apply_spooled_messages
>
> @@ -2024,6 +2141,21 @@ apply_spooled_messages(FileSet *stream_fileset,
> TransactionId xid,
> int fileno;
> off_t offset;
>
> + /*
> + * Should we delay the current transaction?
> + *
> + * Unlike the regular (non-streamed) cases, the delay is applied in a
> + * STREAM COMMIT/STREAM PREPARE message for streamed transactions.
> The
> + * STREAM START message does not contain a commit/prepare time (it will
> + be
> + * available when the in-progress transaction finishes). Hence, it's
> + not
> + * appropriate to apply a delay at that time.
> + *
> + * It's not allowed to execute time-delayed replication with parallel
> + * apply feature.
> + */
> + if (!am_parallel_apply_worker())
> + maybe_delay_apply(finish_ts);
>
> That whole comment part "Unlike the regular (non-streamed) cases"
> seems misplaced here. Perhaps this part of the comment is better put into
> the function header where the meaning of 'finish_ts' is explained?
Moved it to the header comment for maybe_delay_apply.
> ~~~
>
> 18. apply_spooled_messages
>
> + * It's not allowed to execute time-delayed replication with parallel
> + * apply feature.
> + */
> + if (!am_parallel_apply_worker())
> + maybe_delay_apply(finish_ts);
>
> As was mentioned in comment #11 above this code could be changed like
>
> if (finish_ts)
> maybe_delay_apply(finish_ts);
> then you don't even need to make mention of "parallel apply" at all here.
>
> OTOH if you want to still have the parallel apply comment then maybe reword it
> like this:
> "It is not allowed to combine time-delayed replication with the parallel apply
> feature."
Changed and now I don't mention the parallel apply feature.
> ~~~
>
> 19. apply_spooled_messages
>
> If you chose not to do my suggestion from comment #11, then there are
> 2 identical conditions (!am_parallel_apply_worker()); In this case, I was
> wondering if it would be better to refactor to use a single condition instead.
I applied #11 comment. Now, the conditions are not identical.
> ~~~
>
> 20. send_feedback
> (same as comment #13)
>
> Maybe change the new param name to “in_delayed_apply”?
Changed.
> ~~~
>
> 21.
>
> @@ -3737,8 +3869,15 @@ send_feedback(XLogRecPtr recvpos, bool force,
> bool requestReply)
> /*
> * No outstanding transactions to flush, we can report the latest received
> * position. This is important for synchronous replication.
> + *
> + * During the delay of time-delayed replication, do not tell the
> + publisher
> + * that the received latest LSN is already applied and flushed at this
> + * stage, since we don't apply the transaction yet. If we do so, it
> + leads
> + * to a wrong assumption of logical replication progress on the
> + publisher
> + * side. Here, we just send a feedback message to avoid publisher's
> + * timeout during the delay.
> */
>
> Minor rewording of the comment
>
> SUGGESTION
> If the subscriber side apply is delayed (because of time-delayed
> replication) then do not tell the publisher that the received latest LSN is already
> applied and flushed, otherwise, it leads to the publisher side making a wrong
> assumption of logical replication progress. Instead, we just send a feedback
> message to avoid a publisher timeout during the delay.
Adopted.
> ======
>
>
> src/bin/pg_dump/pg_dump.c
>
> 22.
>
> @@ -4546,9 +4547,14 @@ getSubscriptions(Archive *fout)
> LOGICALREP_TWOPHASE_STATE_DISABLED);
>
> if (fout->remoteVersion >= 160000)
> - appendPQExpBufferStr(query, " s.suborigin\n");
> + appendPQExpBufferStr(query,
> + " s.suborigin,\n"
> + " s.subminapplydelay\n");
> else
> - appendPQExpBuffer(query, " '%s' AS suborigin\n",
> LOGICALREP_ORIGIN_ANY);
> + {
> + appendPQExpBuffer(query, " '%s' AS suborigin,\n",
> + LOGICALREP_ORIGIN_ANY); appendPQExpBufferStr(query, " 0 AS
> + subminapplydelay\n"); }
>
> Can’t those appends in the else part can be combined to a single
> appendPQExpBuffer
>
> appendPQExpBuffer(query,
> " '%s' AS suborigin,\n"
> " 0 AS subminapplydelay\n"
> LOGICALREP_ORIGIN_ANY);
Adopted.
> ======
>
> src/include/catalog/pg_subscription.h
>
> 23.
>
> @@ -70,6 +70,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
> BKI_SHARED_RELATION BKI_ROW
> XLogRecPtr subskiplsn; /* All changes finished at this LSN are
> * skipped */
>
> + int64 subminapplydelay; /* Replication apply delay */
> +
> NameData subname; /* Name of the subscription */
>
> Oid subowner BKI_LOOKUP(pg_authid); /* Owner of the subscription */
>
> SUGGESTION (for comment)
> Replication apply delay (ms)
Fixed.
> ~~
>
> 24.
>
> @@ -120,6 +122,7 @@ typedef struct Subscription
> * in */
> XLogRecPtr skiplsn; /* All changes finished at this LSN are
> * skipped */
> + int64 minapplydelay; /* Replication apply delay */
>
> SUGGESTION (for comment)
> Replication apply delay (ms)
Fixed.
Kindly have a look at the latest v17 patch in [1].
[1] - https://www.postgresql.org/message-id/TYCPR01MB8373F5162C7A0E6224670CF0EDC49%40TYCPR01MB8373.jpnprd0...
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-19 08:16 Takamichi Osumi (Fujitsu) <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-19 08:16 UTC (permalink / raw)
To: 'Peter Smith' <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Thursday, January 19, 2023 10:42 AM Peter Smith <[email protected]> wrote:
> On Wed, Jan 18, 2023 at 6:06 PM Peter Smith <[email protected]>
> wrote:
> >
> > Here are my review comments for the latest patch v16-0001. (excluding
> > the test code)
> >
> ...
> >
> > 8. AlterSubscription (general)
> >
> > I observed during testing there are 3 different errors….
> >
> > At subscription CREATE time you can get this error:
> > ERROR: min_apply_delay > 0 and streaming = parallel are mutually
> > exclusive options
> >
> > If you try to ALTER the min_apply_delay when already streaming =
> > parallel you can get this error:
> > ERROR: cannot enable min_apply_delay for subscription in streaming =
> > parallel mode
> >
> > If you try to ALTER the streaming to be parallel if there is already a
> > min_apply_delay > 0 then you can get this error:
> > ERROR: cannot enable streaming = parallel mode for subscription with
> > min_apply_delay
> >
> > ~
> >
> > IMO there is no need to have 3 different error message texts. I think
> > all these cases are explained by just the first text (ERROR:
> > min_apply_delay > 0 and streaming = parallel are mutually exclusive
> > options)
> >
> >
>
> After checking the regression test output I can see the merit of your separate
> error messages like this, even if they are maybe not strictly necessary. So feel
> free to ignore my previous review comment.
Thank you for your notification.
I wrote another reason why we wrote those messages in [1].
So, please have a look at it.
[1] - https://www.postgresql.org/message-id/TYCPR01MB8373447440202B248BB63805EDC49%40TYCPR01MB8373.jpnprd0...
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-19 10:54 vignesh C <[email protected]>
parent: Takamichi Osumi (Fujitsu) <[email protected]>
2 siblings, 2 replies; 40+ messages in thread
From: vignesh C @ 2023-01-19 10:54 UTC (permalink / raw)
To: Takamichi Osumi (Fujitsu) <[email protected]>; +Cc: Peter Smith <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Thu, 19 Jan 2023 at 12:06, Takamichi Osumi (Fujitsu)
<[email protected]> wrote:
>
> Updated the comment and the function call.
>
> Kindly have a look at the updated patch v17.
Thanks for the updated patch, few comments:
1) min_apply_delay was accepting values like '600 m s h', I was not
sure if we should allow this:
alter subscription sub1 set (min_apply_delay = ' 600 m s h');
+ /*
+ * If no unit was specified, then explicitly
add 'ms' otherwise
+ * the interval_in function would assume 'seconds'.
+ */
+ if (strspn(tmp, "-0123456789 ") == strlen(tmp))
+ val = psprintf("%sms", tmp);
+ else
+ val = tmp;
+
+ interval =
DatumGetIntervalP(DirectFunctionCall3(interval_in,
+
CStringGetDatum(val),
+
ObjectIdGetDatum(InvalidOid),
+
Int32GetDatum(-1)));
2) How about adding current_txn_wait_time in
pg_stat_subscription_stats, we can update the current_txn_wait_time
periodically, this will help the user to check approximately how much
time is left(min_apply_delay - stat value) before this transaction
will be applied in the subscription. If you agree this can be 0002
patch.
3) There is one check at parse_subscription_options and another check
in AlterSubscription, this looks like a redundant check in case of
alter subscription, can we try to merge and keep in one place:
/*
* The combination of parallel streaming mode and min_apply_delay is not
* allowed.
*/
if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
opts->min_apply_delay > 0)
{
if (opts->streaming == LOGICALREP_STREAM_PARALLEL)
ereport(ERROR,
errcode(ERRCODE_SYNTAX_ERROR),
errmsg("%s and %s are mutually exclusive options",
"min_apply_delay > 0", "streaming = parallel"));
}
if (IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY))
{
/*
* The combination of parallel streaming mode and
* min_apply_delay is not allowed.
*/
if (opts.min_apply_delay > 0)
if ((IsSet(opts.specified_opts, SUBOPT_STREAMING) && opts.streaming ==
LOGICALREP_STREAM_PARALLEL) ||
(!IsSet(opts.specified_opts, SUBOPT_STREAMING) && sub->stream ==
LOGICALREP_STREAM_PARALLEL))
ereport(ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("cannot enable %s for subscription in %s mode",
"min_apply_delay", "streaming = parallel"));
values[Anum_pg_subscription_subminapplydelay - 1] =
Int64GetDatum(opts.min_apply_delay);
replaces[Anum_pg_subscription_subminapplydelay - 1] = true;
}
4) typo "execeeds" should be "exceeds"
+ time on the subscriber. Any overhead of time spent in
logical decoding
+ and in transferring the transaction may reduce the actual wait time.
+ It is also possible that the overhead already execeeds the requested
+ <literal>min_apply_delay</literal> value, in which case no additional
+ wait is necessary. If the system clocks on publisher and subscriber
+ are not synchronized, this may lead to apply changes earlier than
Regards,
Vignesh
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-19 12:59 Amit Kapila <[email protected]>
parent: vignesh C <[email protected]>
1 sibling, 1 reply; 40+ messages in thread
From: Amit Kapila @ 2023-01-19 12:59 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Takamichi Osumi (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Thu, Jan 19, 2023 at 4:25 PM vignesh C <[email protected]> wrote:
>
> On Thu, 19 Jan 2023 at 12:06, Takamichi Osumi (Fujitsu)
> <[email protected]> wrote:
> >
> > Updated the comment and the function call.
> >
> > Kindly have a look at the updated patch v17.
>
> Thanks for the updated patch, few comments:
> 1) min_apply_delay was accepting values like '600 m s h', I was not
> sure if we should allow this:
> alter subscription sub1 set (min_apply_delay = ' 600 m s h');
>
I think here we should have specs similar to recovery_min_apply_delay.
>
> 2) How about adding current_txn_wait_time in
> pg_stat_subscription_stats, we can update the current_txn_wait_time
> periodically, this will help the user to check approximately how much
> time is left(min_apply_delay - stat value) before this transaction
> will be applied in the subscription. If you agree this can be 0002
> patch.
>
Do we have any similar stats for recovery_min_apply_delay? If not, I
suggest let's postpone this to see if users really need such a
parameter.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-19 13:17 Amit Kapila <[email protected]>
parent: Takamichi Osumi (Fujitsu) <[email protected]>
2 siblings, 1 reply; 40+ messages in thread
From: Amit Kapila @ 2023-01-19 13:17 UTC (permalink / raw)
To: Takamichi Osumi (Fujitsu) <[email protected]>; +Cc: Peter Smith <[email protected]>; vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Thu, Jan 19, 2023 at 12:06 PM Takamichi Osumi (Fujitsu)
<[email protected]> wrote:
>
> Kindly have a look at the updated patch v17.
>
Can we try to optimize the test time for this test? On my machine, it
is the second highest time-consuming test in src/test/subscription. It
seems you are waiting twice for apply_delay and both are for streaming
cases by varying the number of changes. I think it should be just once
and that too for the non-streaming case. I think it would be good to
test streaming code path interaction but not sure if it is important
enough to have two test cases for apply_delay.
One minor comment that I observed while going through the patch.
+ /*
+ * The combination of parallel streaming mode and min_apply_delay is not
+ * allowed.
+ */
+ if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ opts->min_apply_delay > 0)
I think it would be good if you can specify the reason for not
allowing this combination in the comments.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-19 13:23 vignesh C <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: vignesh C @ 2023-01-19 13:23 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Takamichi Osumi (Fujitsu) <[email protected]>; Peter Smith <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Thu, 19 Jan 2023 at 18:29, Amit Kapila <[email protected]> wrote:
>
> On Thu, Jan 19, 2023 at 4:25 PM vignesh C <[email protected]> wrote:
> >
> > On Thu, 19 Jan 2023 at 12:06, Takamichi Osumi (Fujitsu)
> > <[email protected]> wrote:
> > >
> > > Updated the comment and the function call.
> > >
> > > Kindly have a look at the updated patch v17.
> >
> > Thanks for the updated patch, few comments:
> > 1) min_apply_delay was accepting values like '600 m s h', I was not
> > sure if we should allow this:
> > alter subscription sub1 set (min_apply_delay = ' 600 m s h');
> >
>
> I think here we should have specs similar to recovery_min_apply_delay.
>
> >
> > 2) How about adding current_txn_wait_time in
> > pg_stat_subscription_stats, we can update the current_txn_wait_time
> > periodically, this will help the user to check approximately how much
> > time is left(min_apply_delay - stat value) before this transaction
> > will be applied in the subscription. If you agree this can be 0002
> > patch.
> >
>
> Do we have any similar stats for recovery_min_apply_delay? If not, I
> suggest let's postpone this to see if users really need such a
> parameter.
I did not find any statistics for recovery_min_apply_delay, ok it can
be delayed to a later time.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-20 03:46 shveta malik <[email protected]>
parent: Takamichi Osumi (Fujitsu) <[email protected]>
0 siblings, 2 replies; 40+ messages in thread
From: shveta malik @ 2023-01-20 03:46 UTC (permalink / raw)
To: Takamichi Osumi (Fujitsu) <[email protected]>; +Cc: Peter Smith <[email protected]>; vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; shveta malik <[email protected]>
On Thu, Jan 19, 2023 at 12:42 PM Takamichi Osumi (Fujitsu)
<[email protected]> wrote:
>
> On Wednesday, January 18, 2023 4:06 PM Peter Smith <[email protected]> wrote:
> > Here are my review comments for the latest patch v16-0001. (excluding the
> > test code)
> Hi, thank you for your review !
>
> > ======
> >
> > General
> >
> > 1.
> >
> > Since the value of min_apply_delay cannot be < 0, I was thinking probably it
> > should have been declared everywhere in this patch as a
> > uint64 instead of an int64, right?
> No, we won't be able to adopt this idea.
>
> It seems that we are not able to use uint for catalog type.
> So, can't applying it to the pg_subscription.h definitions
> and then similarly Int64GetDatum to store catalog variables
> and the argument variable of Int64GetDatum.
>
> Plus, there is a possibility that type Interval becomes negative value,
> then we are not able to change the int64 variable to get
> the return value of interval2ms().
>
> > ======
> >
> > Commit message
> >
> > 2.
> >
> > If the subscription sets min_apply_delay parameter, the logical replication
> > worker will delay the transaction commit for min_apply_delay milliseconds.
> >
> > ~
> >
> > IMO there should be another sentence before this just to say that a new
> > parameter is being added:
> >
> > e.g.
> > This patch implements a new subscription parameter called
> > 'min_apply_delay'.
> Added.
>
>
> > ======
> >
> > doc/src/sgml/config.sgml
> >
> > 3.
> >
> > + <para>
> > + For time-delayed logical replication, the apply worker sends a Standby
> > + Status Update message to the corresponding publisher per the
> > indicated
> > + time of this parameter. Therefore, if this parameter is longer than
> > + <literal>wal_sender_timeout</literal> on the publisher, then the
> > + walsender doesn't get any update message during the delay and
> > repeatedly
> > + terminates due to the timeout errors. Hence, make sure this parameter
> > is
> > + shorter than the <literal>wal_sender_timeout</literal> of the
> > publisher.
> > + If this parameter is set to zero with time-delayed replication, the
> > + apply worker doesn't send any feedback messages during the
> > + <literal>min_apply_delay</literal>.
> > + </para>
> >
> >
> > This paragraph seemed confusing. I think it needs to be reworded to change all
> > of the "this parameter" references because there are at least 3 different
> > parameters mentioned in this paragraph. e.g. maybe just change them to
> > explicitly name the parameter you are talking about.
> >
> > I also think it needs to mention the ‘min_apply_delay’ subscription parameter
> > up-front and then refer to it appropriately.
> >
> > The end result might be something like I wrote below (this is just my guess ?
> > probably you can word it better).
> >
> > SUGGESTION
> > For time-delayed logical replication (i.e. when the subscription is created with
> > parameter min_apply_delay > 0), the apply worker sends a Standby Status
> > Update message to the publisher with a period of wal_receiver_status_interval .
> > Make sure to set wal_receiver_status_interval less than the
> > wal_sender_timeout on the publisher, otherwise, the walsender will repeatedly
> > terminate due to the timeout errors. If wal_receiver_status_interval is set to zero,
> > the apply worker doesn't send any feedback messages during the subscriber’s
> > min_apply_delay period.
> Applied. Also, I added one reference for min_apply_delay parameter
> at the end of this description.
>
>
> > ======
> >
> > doc/src/sgml/ref/create_subscription.sgml
> >
> > 4.
> >
> > + <para>
> > + By default, the subscriber applies changes as soon as possible. As
> > + with the physical replication feature
> > + (<xref linkend="guc-recovery-min-apply-delay"/>), it can be
> > useful to
> > + have a time-delayed logical replica. This parameter lets the user to
> > + delay the application of changes by a specified amount of
> > time. If this
> > + value is specified without units, it is taken as milliseconds. The
> > + default is zero(no delay).
> > + </para>
> >
> > 4a.
> > As with the physical replication feature (recovery_min_apply_delay), it can be
> > useful to have a time-delayed logical replica.
> >
> > IMO not sure that the above sentence is necessary. It seems only to be saying
> > that this parameter can be useful. Why do we need to say that?
> Removed the sentence.
>
>
> > ~
> >
> > 4b.
> > "This parameter lets the user to delay" -> "This parameter lets the user delay"
> > OR
> > "This parameter lets the user to delay" -> "This parameter allows the user to
> > delay"
> Fixed.
>
>
> > ~
> >
> > 4c.
> > "If this value is specified without units" -> "If the value is specified without
> > units"
> Fixed.
>
> > ~
> >
> > 4d.
> > "zero(no delay)." -> "zero (no delay)."
> Fixed.
>
> > ----
> >
> > 5.
> >
> > + <para>
> > + The delay occurs only on WAL records for transaction begins and
> > after
> > + the initial table synchronization. It is possible that the
> > + replication delay between publisher and subscriber exceeds the
> > value
> > + of this parameter, in which case no delay is added. Note that the
> > + delay is calculated between the WAL time stamp as written on
> > + publisher and the current time on the subscriber. Time
> > spent in logical
> > + decoding and in transferring the transaction may reduce the
> > actual wait
> > + time. If the system clocks on publisher and subscriber are not
> > + synchronized, this may lead to apply changes earlier than
> > expected,
> > + but this is not a major issue because this parameter is
> > typically much
> > + larger than the time deviations between servers. Note that if this
> > + parameter is set to a long delay, the replication will stop if the
> > + replication slot falls behind the current LSN by more than
> > + <link
> > linkend="guc-max-slot-wal-keep-size"><literal>max_slot_wal_keep_size</
> > literal></link>.
> > + </para>
> >
> > I think the first part can be reworded slightly. See what you think about the
> > suggestion below.
> >
> > SUGGESTION
> > Any delay occurs only on WAL records for transaction begins after all initial
> > table synchronization has finished. The delay is calculated between the WAL
> > timestamp as written on the publisher and the current time on the subscriber.
> > Any overhead of time spent in logical decoding and in transferring the
> > transaction may reduce the actual wait time.
> > It is also possible that the overhead already exceeds the requested
> > 'min_apply_delay' value, in which case no additional wait is necessary. If the
> > system clocks...
> Addressed.
>
>
> > ----
> >
> > 6.
> >
> > + <para>
> > + Setting streaming to <literal>parallel</literal> mode and
> > <literal>min_apply_delay</literal>
> > + simultaneously is not supported.
> > + </para>
> >
> > SUGGESTION
> > A non-zero min_apply_delay parameter is not allowed when streaming in
> > parallel mode.
> Applied.
>
>
> > ======
> >
> > src/backend/commands/subscriptioncmds.c
> >
> > 7. parse_subscription_options
> >
> > @@ -404,6 +445,17 @@ parse_subscription_options(ParseState *pstate, List
> > *stmt_options,
> > "slot_name = NONE", "create_slot = false")));
> > }
> > }
> > +
> > + /* Test the combination of streaming mode and min_apply_delay */ if
> > + (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
> > + opts->min_apply_delay > 0)
> > + {
> > + if (opts->streaming == LOGICALREP_STREAM_PARALLEL)
> > ereport(ERROR,
> > + errcode(ERRCODE_SYNTAX_ERROR), errmsg("%s and %s are mutually
> > + exclusive options",
> > + "min_apply_delay > 0", "streaming = parallel")); }
> >
> > SUGGESTION (comment)
> > The combination of parallel streaming mode and min_apply_delay is not
> > allowed.
> Fixed.
>
>
> > ~~~
> >
> > 8. AlterSubscription (general)
> >
> > I observed during testing there are 3 different errors….
> >
> > At subscription CREATE time you can get this error:
> > ERROR: min_apply_delay > 0 and streaming = parallel are mutually exclusive
> > options
> >
> > If you try to ALTER the min_apply_delay when already streaming = parallel you
> > can get this error:
> > ERROR: cannot enable min_apply_delay for subscription in streaming =
> > parallel mode
> >
> > If you try to ALTER the streaming to be parallel if there is already a
> > min_apply_delay > 0 then you can get this error:
> > ERROR: cannot enable streaming = parallel mode for subscription with
> > min_apply_delay
> Yes. This is because the existing error message styles
> in AlterSubscription and parse_subscription_options.
>
> The former uses "mutually exclusive" messages consistently,
> while the latter does "cannot enable ..." ones.
> > ~
> >
> > IMO there is no need to have 3 different error message texts. I think all these
> > cases are explained by just the first text (ERROR:
> > min_apply_delay > 0 and streaming = parallel are mutually exclusive
> > options)
> Then, we followed this kind of formats.
>
>
> > ~~~
> >
> > 9. AlterSubscription
> >
> > @@ -1098,6 +1152,18 @@ AlterSubscription(ParseState *pstate,
> > AlterSubscriptionStmt *stmt,
> >
> > if (IsSet(opts.specified_opts, SUBOPT_STREAMING))
> > {
> > + /*
> > + * Test the combination of streaming mode and
> > + * min_apply_delay
> > + */
> > + if (opts.streaming == LOGICALREP_STREAM_PARALLEL) if
> > + ((IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) &&
> > opts.min_apply_delay > 0) ||
> > + (!IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) &&
> > sub->minapplydelay > 0))
> > + ereport(ERROR,
> > + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> > + errmsg("cannot enable %s mode for subscription with %s",
> > + "streaming = parallel", "min_apply_delay"));
> > +
> >
> > 9a.
> > SUGGESTION (comment)
> > The combination of parallel streaming mode and min_apply_delay is not
> > allowed.
> Fixed.
>
>
> > ~
> >
> > 9b.
> > (see AlterSubscription general review comment #8 above) Here you can use the
> > same comment error message that says min_apply_delay > 0 and streaming =
> > parallel are mutually exclusive options.
> As described above, we followed the current style in the existing functions.
>
>
> > ~~~
> >
> > 10. AlterSubscription
> >
> > @@ -1111,6 +1177,25 @@ AlterSubscription(ParseState *pstate,
> > AlterSubscriptionStmt *stmt,
> > = true;
> > }
> >
> > + if (IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY)) {
> > + /*
> > + * Test the combination of streaming mode and
> > + * min_apply_delay
> > + */
> > + if (opts.min_apply_delay > 0)
> > + if ((IsSet(opts.specified_opts, SUBOPT_STREAMING) && opts.streaming
> > == LOGICALREP_STREAM_PARALLEL) ||
> > + (!IsSet(opts.specified_opts, SUBOPT_STREAMING) && sub->stream ==
> > LOGICALREP_STREAM_PARALLEL))
> > + ereport(ERROR,
> > + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> > + errmsg("cannot enable %s for subscription in %s mode",
> > + "min_apply_delay", "streaming = parallel"));
> > +
> > + values[Anum_pg_subscription_subminapplydelay - 1] =
> > + Int64GetDatum(opts.min_apply_delay);
> > + replaces[Anum_pg_subscription_subminapplydelay - 1] = true; }
> >
> > 10a.
> > SUGGESTION (comment)
> > The combination of parallel streaming mode and min_apply_delay is not
> > allowed.
> Fixed.
>
>
> > ~
> >
> > 10b.
> > (see AlterSubscription general review comment #8 above) Here you can use the
> > same comment error message that says min_apply_delay > 0 and streaming =
> > parallel are mutually exclusive options.
> Same as 9b.
>
> > ======
> >
> > .../replication/logical/applyparallelworker.c
> >
> > 11.
> >
> > @@ -704,7 +704,8 @@ pa_process_spooled_messages_if_required(void)
> > {
> > apply_spooled_messages(&MyParallelShared->fileset,
> > MyParallelShared->xid,
> > - InvalidXLogRecPtr);
> > + InvalidXLogRecPtr,
> > + 0);
> >
> > IMO this passing of 0 is a bit strange because it is currently acting like a dummy
> > value since the apply_spooled_messages will never make use of the 'finish_ts'
> > anyway (since this call is from a parallel apply worker).
> >
> > I think a better way to code this might be to pass the 0 (same as you are doing
> > here) but inside the apply_spooled_messages change the code:
> >
> > FROM
> > if (!am_parallel_apply_worker())
> > maybe_delay_apply(finish_ts);
> >
> > TO
> > if (finish_ts)
> > maybe_delay_apply(finish_ts);
> >
> > That does 2 things.
> > - It makes the passed-in 0 have some meaning
> > - It simplifies the apply_spooled_messages code
> Adopted.
>
>
> > ======
> >
> > src/backend/replication/logical/worker.c
> >
> > 12.
> >
> > @@ -318,6 +318,17 @@ static List *on_commit_wakeup_workers_subids =
> > NIL; bool in_remote_transaction = false; static XLogRecPtr
> > remote_final_lsn = InvalidXLogRecPtr;
> >
> > +/*
> > + * In order to avoid walsender's timeout during time-delayed
> > +replication,
> > + * it's necessary to keep sending feedback messages during the delay
> > +from the
> > + * worker process. Meanwhile, the feature delays the apply before
> > +starting the
> > + * transaction and thus we don't write WALs for the suspended changes
> > +during
> > + * the wait. Hence, in the case the worker process sends a feedback
> > +message
> > + * during the delay, we should not make positions of the flushed and
> > +apply LSN
> > + * overwritten by the last received latest LSN. See send_feedback()
> > for details.
> > + */
> > +static XLogRecPtr last_received = InvalidXLogRecPtr;
> >
> > 12a.
> > Suggest a small change to the first sentence of the comment.
> >
> > BEFORE
> > In order to avoid walsender's timeout during time-delayed replication, it's
> > necessary to keep sending feedback messages during the delay from the
> > worker process.
> >
> > AFTER
> > In order to avoid walsender timeout for time-delayed replication the worker
> > process keeps sending feedback messages during the delay period.
> Fixed.
>
>
> > ~
> >
> > 12b.
> > "Hence, in the case" -> "When"
> Fixed.
>
>
> > ~~~
> >
> > 13. forward declare
> >
> > -static void send_feedback(XLogRecPtr recvpos, bool force, bool
> > requestReply);
> > +static void send_feedback(XLogRecPtr recvpos, bool force, bool
> > requestReply,
> > + bool in_delaying_apply);
> >
> > Change the param name:
> >
> > "in_delaying_apply" -> "in_delayed_apply” (??)
> Changed. The initial intention to append the "in_"
> prefix is to make the variable name aligned with
> some other variables such as "in_remote_transaction" and
> "in_streamed_transaction" that mean the current status
> for the transaction. So, until there is a better name proposed,
> we can keep it.
>
>
> > ~~~
> >
> > 14. maybe_delay_apply
> >
> > + /* Nothing to do if no delay set */
> > + if (MySubscription->minapplydelay <= 0) return;
> >
> > IIUC min_apply_delay cannot be < 0 so this condition could simply be:
> >
> > if (!MySubscription->minapplydelay)
> > return;
> Fixed.
>
>
> > ~~~
> >
> > 15. maybe_delay_apply
> >
> > + /*
> > + * The min_apply_delay parameter is ignored until all tablesync workers
> > + * have reached READY state. If we allow the delay during the catchup
> > + * phase, once we reach the limit of tablesync workers, it will impose
> > + a
> > + * delay for each subsequent worker. It means it will take a long time
> > + to
> > + * finish the initial table synchronization.
> > + */
> > + if (!AllTablesyncsReady())
> > + return;
> >
> > SUGGESTION (slight rewording)
> > The min_apply_delay parameter is ignored until all tablesync workers have
> > reached READY state. This is because if we allowed the delay during the
> > catchup phase, then once we reached the limit of tablesync workers it would
> > impose a delay for each subsequent worker. That would cause initial table
> > synchronization completion to take a long time.
> Fixed.
>
>
> > ~~~
> >
> > 16. maybe_delay_apply
> >
> > + while (true)
> > + {
> > + long diffms;
> > +
> > + ResetLatch(MyLatch);
> > +
> > + CHECK_FOR_INTERRUPTS();
> >
> > IMO there should be some small explanatory comment here at the top of the
> > while loop.
> Added.
>
>
> > ~~~
> >
> > 17. apply_spooled_messages
> >
> > @@ -2024,6 +2141,21 @@ apply_spooled_messages(FileSet *stream_fileset,
> > TransactionId xid,
> > int fileno;
> > off_t offset;
> >
> > + /*
> > + * Should we delay the current transaction?
> > + *
> > + * Unlike the regular (non-streamed) cases, the delay is applied in a
> > + * STREAM COMMIT/STREAM PREPARE message for streamed transactions.
> > The
> > + * STREAM START message does not contain a commit/prepare time (it will
> > + be
> > + * available when the in-progress transaction finishes). Hence, it's
> > + not
> > + * appropriate to apply a delay at that time.
> > + *
> > + * It's not allowed to execute time-delayed replication with parallel
> > + * apply feature.
> > + */
> > + if (!am_parallel_apply_worker())
> > + maybe_delay_apply(finish_ts);
> >
> > That whole comment part "Unlike the regular (non-streamed) cases"
> > seems misplaced here. Perhaps this part of the comment is better put into
> > the function header where the meaning of 'finish_ts' is explained?
> Moved it to the header comment for maybe_delay_apply.
>
>
> > ~~~
> >
> > 18. apply_spooled_messages
> >
> > + * It's not allowed to execute time-delayed replication with parallel
> > + * apply feature.
> > + */
> > + if (!am_parallel_apply_worker())
> > + maybe_delay_apply(finish_ts);
> >
> > As was mentioned in comment #11 above this code could be changed like
> >
> > if (finish_ts)
> > maybe_delay_apply(finish_ts);
> > then you don't even need to make mention of "parallel apply" at all here.
> >
> > OTOH if you want to still have the parallel apply comment then maybe reword it
> > like this:
> > "It is not allowed to combine time-delayed replication with the parallel apply
> > feature."
> Changed and now I don't mention the parallel apply feature.
>
> > ~~~
> >
> > 19. apply_spooled_messages
> >
> > If you chose not to do my suggestion from comment #11, then there are
> > 2 identical conditions (!am_parallel_apply_worker()); In this case, I was
> > wondering if it would be better to refactor to use a single condition instead.
> I applied #11 comment. Now, the conditions are not identical.
>
> > ~~~
> >
> > 20. send_feedback
> > (same as comment #13)
> >
> > Maybe change the new param name to “in_delayed_apply”?
> Changed.
>
>
> > ~~~
> >
> > 21.
> >
> > @@ -3737,8 +3869,15 @@ send_feedback(XLogRecPtr recvpos, bool force,
> > bool requestReply)
> > /*
> > * No outstanding transactions to flush, we can report the latest received
> > * position. This is important for synchronous replication.
> > + *
> > + * During the delay of time-delayed replication, do not tell the
> > + publisher
> > + * that the received latest LSN is already applied and flushed at this
> > + * stage, since we don't apply the transaction yet. If we do so, it
> > + leads
> > + * to a wrong assumption of logical replication progress on the
> > + publisher
> > + * side. Here, we just send a feedback message to avoid publisher's
> > + * timeout during the delay.
> > */
> >
> > Minor rewording of the comment
> >
> > SUGGESTION
> > If the subscriber side apply is delayed (because of time-delayed
> > replication) then do not tell the publisher that the received latest LSN is already
> > applied and flushed, otherwise, it leads to the publisher side making a wrong
> > assumption of logical replication progress. Instead, we just send a feedback
> > message to avoid a publisher timeout during the delay.
> Adopted.
>
>
> > ======
> >
> >
> > src/bin/pg_dump/pg_dump.c
> >
> > 22.
> >
> > @@ -4546,9 +4547,14 @@ getSubscriptions(Archive *fout)
> > LOGICALREP_TWOPHASE_STATE_DISABLED);
> >
> > if (fout->remoteVersion >= 160000)
> > - appendPQExpBufferStr(query, " s.suborigin\n");
> > + appendPQExpBufferStr(query,
> > + " s.suborigin,\n"
> > + " s.subminapplydelay\n");
> > else
> > - appendPQExpBuffer(query, " '%s' AS suborigin\n",
> > LOGICALREP_ORIGIN_ANY);
> > + {
> > + appendPQExpBuffer(query, " '%s' AS suborigin,\n",
> > + LOGICALREP_ORIGIN_ANY); appendPQExpBufferStr(query, " 0 AS
> > + subminapplydelay\n"); }
> >
> > Can’t those appends in the else part can be combined to a single
> > appendPQExpBuffer
> >
> > appendPQExpBuffer(query,
> > " '%s' AS suborigin,\n"
> > " 0 AS subminapplydelay\n"
> > LOGICALREP_ORIGIN_ANY);
> Adopted.
>
>
> > ======
> >
> > src/include/catalog/pg_subscription.h
> >
> > 23.
> >
> > @@ -70,6 +70,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
> > BKI_SHARED_RELATION BKI_ROW
> > XLogRecPtr subskiplsn; /* All changes finished at this LSN are
> > * skipped */
> >
> > + int64 subminapplydelay; /* Replication apply delay */
> > +
> > NameData subname; /* Name of the subscription */
> >
> > Oid subowner BKI_LOOKUP(pg_authid); /* Owner of the subscription */
> >
> > SUGGESTION (for comment)
> > Replication apply delay (ms)
> Fixed.
>
> > ~~
> >
> > 24.
> >
> > @@ -120,6 +122,7 @@ typedef struct Subscription
> > * in */
> > XLogRecPtr skiplsn; /* All changes finished at this LSN are
> > * skipped */
> > + int64 minapplydelay; /* Replication apply delay */
> >
> > SUGGESTION (for comment)
> > Replication apply delay (ms)
> Fixed.
>
>
> Kindly have a look at the latest v17 patch in [1].
>
>
> [1] - https://www.postgresql.org/message-id/TYCPR01MB8373F5162C7A0E6224670CF0EDC49%40TYCPR01MB8373.jpnprd0...
>
> Best Regards,
> Takamichi Osumi
>
1)
Tried different variations of altering 'min_apply_delay'. All passed
except one below:
postgres=# alter subscription mysubnew set (min_apply_delay = '10.9min 1ms');
ALTER SUBSCRIPTION
postgres=# alter subscription mysubnew set (min_apply_delay = '10.9min 2s 1ms');
ALTER SUBSCRIPTION
--very similar to above but fails,
postgres=# alter subscription mysubnew set (min_apply_delay = '10.9s 1ms');
ERROR: invalid input syntax for type interval: "10.9s 1ms"
2)
Logging:
2023-01-19 17:33:16.202 IST [404797] DEBUG: logical replication apply
delay: 19979 ms
2023-01-19 17:33:26.212 IST [404797] DEBUG: logical replication apply
delay: 9969 ms
2023-01-19 17:34:25.730 IST [404962] DEBUG: logical replication apply
delay: 179988 ms-->previous wait over, started for next txn
2023-01-19 17:34:35.737 IST [404962] DEBUG: logical replication apply
delay: 169981 ms
2023-01-19 17:34:45.746 IST [404962] DEBUG: logical replication apply
delay: 159972 ms
Is there a way to distinguish between these logs? Maybe dumping xids along-with?
thanks
Shveta
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-20 06:55 Peter Smith <[email protected]>
parent: Takamichi Osumi (Fujitsu) <[email protected]>
2 siblings, 1 reply; 40+ messages in thread
From: Peter Smith @ 2023-01-20 06:55 UTC (permalink / raw)
To: Takamichi Osumi (Fujitsu) <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
Hi Osumi-san, here are my review comments for the latest patch v17-0001.
======
Commit Message
1.
Prohibit the combination of this feature and parallel streaming mode.
SUGGESTION (using the same wording as in the code comments)
The combination of parallel streaming mode and min_apply_delay is not allowed.
======
doc/src/sgml/ref/create_subscription.sgml
2.
+ <para>
+ By default, the subscriber applies changes as soon as possible. This
+ parameter allows the user to delay the application of changes by a
+ specified amount of time. If the value is specified without units, it
+ is taken as milliseconds. The default is zero (no delay).
+ </para>
Looking at this again, it seemed a bit strange to repeat "specified"
twice in 2 sentences. Maybe change one of them.
I’ve also suggested using the word "interval" because I don’t think
docs yet mentioned anywhere (except in the example) that using
intervals is possible.
SUGGESTION (for the 2nd sentence)
This parameter allows the user to delay the application of changes by
a given time interval.
~~~
3.
+ <para>
+ Any delay occurs only on WAL records for transaction begins after all
+ initial table synchronization has finished. The delay is calculated
+ between the WAL timestamp as written on the publisher and the current
+ time on the subscriber. Any overhead of time spent in
logical decoding
+ and in transferring the transaction may reduce the actual wait time.
+ It is also possible that the overhead already execeeds the requested
+ <literal>min_apply_delay</literal> value, in which case no additional
+ wait is necessary. If the system clocks on publisher and subscriber
+ are not synchronized, this may lead to apply changes earlier than
+ expected, but this is not a major issue because this parameter is
+ typically much larger than the time deviations between servers. Note
+ that if this parameter is set to a long delay, the replication will
+ stop if the replication slot falls behind the current LSN
by more than
+ <link
linkend="guc-max-slot-wal-keep-size"><literal>max_slot_wal_keep_size</literal></link>.
+ </para>
3a.
Typo "execeeds" (I think Vignesh reported this already)
~
3b.
SUGGESTION (for the 2nd sentence)
BEFORE
The delay is calculated between the WAL timestamp...
AFTER
The delay is calculated as the difference between the WAL timestamp...
~~~
4.
+ <warning>
+ <para>
+ Delaying the replication can mean there is a much longer
time between making
+ a change on the publisher, and that change being
committed on the subscriber.
+ v
+ See <xref linkend="guc-synchronous-commit"/>.
+ </para>
+ </warning>
IMO maybe there is a better way to express the 2nd sentence:
BEFORE
This can have a big impact on synchronous replication.
AFTER
This can impact the performance of synchronous replication.
======
src/backend/commands/subscriptioncmds.c
5. parse_subscription_options
@@ -324,6 +328,43 @@ parse_subscription_options(ParseState *pstate,
List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ strcmp(defel->defname, "min_apply_delay") == 0)
+ {
+ char *val,
+ *tmp;
+ Interval *interval;
+ int64 ms;
IMO 'delay_ms' (or similar) would be a friendlier variable name than just 'ms'
~~~
6.
@@ -404,6 +445,20 @@ parse_subscription_options(ParseState *pstate,
List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * The combination of parallel streaming mode and min_apply_delay is not
+ * allowed.
+ */
+ if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ opts->min_apply_delay > 0)
+ {
+ if (opts->streaming == LOGICALREP_STREAM_PARALLEL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s and %s are mutually exclusive options",
+ "min_apply_delay > 0", "streaming = parallel"));
+ }
This could be expressed as a single condition using &&, maybe also
with the brackets eliminated. (Unless you feel the current code is
more readable)
~~~
7.
+ if (opts.min_apply_delay > 0)
+ if ((IsSet(opts.specified_opts, SUBOPT_STREAMING) && opts.streaming
== LOGICALREP_STREAM_PARALLEL) ||
+ (!IsSet(opts.specified_opts, SUBOPT_STREAMING) && sub->stream ==
LOGICALREP_STREAM_PARALLEL))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s for subscription in %s mode",
+ "min_apply_delay", "streaming = parallel"));
These nested ifs could instead be a single "if" with && condition.
(Unless you feel the current code is more readable)
======
src/backend/replication/logical/worker.c
8. maybe_delay_apply
+ * Hence, it's not appropriate to apply a delay at the time.
+ */
+static void
+maybe_delay_apply(TimestampTz finish_ts)
That last sentence "Hence,... delay at the time" does not sound
correct. Is there a typo or missing words here?
Maybe it meant to say "... at the STREAM START time."?
~~~
9.
+ /* This might change wal_receiver_status_interval */
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
I was unsure why did you make a special mention of
'wal_receiver_status_interval' here. I mean, Aren't there also other
GUCs that might change and affect something here so was there some
special reason only this one was mentioned?
======
src/test/subscription/t/032_apply_delay.pl
10.
+
+# Compare inserted time on the publisher with applied time on the subscriber to
+# confirm the latter is applied after expected time.
+sub check_apply_delay_time
Maybe the comment could also mention that the time is automatically
stored in the table column 'c'.
~~~
11.
+# Confirm the suspended record doesn't get applied expectedly by the ALTER
+# DISABLE command.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(a) FROM test_tab WHERE a = 0;");
+is($result, qq(0), "check if the delayed transaction doesn't get
applied expectedly");
The use of "doesn't get applied expectedly" (in 2 places here) seemed
strange. Maybe it's better to say like
SUGGESTION
# Confirm disabling the subscription by ALTER DISABLE did not cause
the delayed transaction to be applied.
$result = $node_subscriber->safe_psql('postgres',
"SELECT count(a) FROM test_tab WHERE a = 0;");
is($result, qq(0), "check the delayed transaction was not applied");
------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-20 07:38 Peter Smith <[email protected]>
parent: shveta malik <[email protected]>
1 sibling, 1 reply; 40+ messages in thread
From: Peter Smith @ 2023-01-20 07:38 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Takamichi Osumi (Fujitsu) <[email protected]>; vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Fri, Jan 20, 2023 at 2:47 PM shveta malik <[email protected]> wrote:
>
...
> 2)
> Logging:
> 2023-01-19 17:33:16.202 IST [404797] DEBUG: logical replication apply
> delay: 19979 ms
> 2023-01-19 17:33:26.212 IST [404797] DEBUG: logical replication apply
> delay: 9969 ms
> 2023-01-19 17:34:25.730 IST [404962] DEBUG: logical replication apply
> delay: 179988 ms-->previous wait over, started for next txn
> 2023-01-19 17:34:35.737 IST [404962] DEBUG: logical replication apply
> delay: 169981 ms
> 2023-01-19 17:34:45.746 IST [404962] DEBUG: logical replication apply
> delay: 159972 ms
>
> Is there a way to distinguish between these logs? Maybe dumping xids along-with?
>
+1
Also, I was thinking of some other logging enhancements
a) the message should say that this is the *remaining* time to left to wait.
b) it might be convenient to know from the log what was the original
min_apply_delay value in the 1st place.
For example, the logs might look something like this:
DEBUG: time-delayed replication for txid 1234, min_apply_delay =
160000 ms. Remaining wait time: 159972 ms
DEBUG: time-delayed replication for txid 1234, min_apply_delay =
160000 ms. Remaining wait time: 142828 ms
DEBUG: time-delayed replication for txid 1234, min_apply_delay =
160000 ms. Remaining wait time: 129994 ms
DEBUG: time-delayed replication for txid 1234, min_apply_delay =
160000 ms. Remaining wait time: 110001 ms
...
------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-20 08:53 shveta malik <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 2 replies; 40+ messages in thread
From: shveta malik @ 2023-01-20 08:53 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Takamichi Osumi (Fujitsu) <[email protected]>; vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; shveta malik <[email protected]>
On Fri, Jan 20, 2023 at 1:08 PM Peter Smith <[email protected]> wrote:
> a) the message should say that this is the *remaining* time to left to wait.
>
> b) it might be convenient to know from the log what was the original
> min_apply_delay value in the 1st place.
>
> For example, the logs might look something like this:
>
> DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> 160000 ms. Remaining wait time: 159972 ms
> DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> 160000 ms. Remaining wait time: 142828 ms
> DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> 160000 ms. Remaining wait time: 129994 ms
> DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> 160000 ms. Remaining wait time: 110001 ms
> ...
>
+1
This will also help when min_apply_delay is set to a new value in
between the current wait. Lets say, I started with min_apply_delay=5
min, when the worker was half way through this, I changed
min_apply_delay to 3 min or say 10min, I see the impact of that change
i.e. new wait-time is adjusted, but log becomes confusing. So, please
keep this scenario as well in mind while improving logging.
thanks
Shveta
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-20 09:13 shveta malik <[email protected]>
parent: shveta malik <[email protected]>
1 sibling, 1 reply; 40+ messages in thread
From: shveta malik @ 2023-01-20 09:13 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Takamichi Osumi (Fujitsu) <[email protected]>; vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; shveta malik <[email protected]>
On Fri, Jan 20, 2023 at 2:23 PM shveta malik <[email protected]> wrote:
>
> On Fri, Jan 20, 2023 at 1:08 PM Peter Smith <[email protected]> wrote:
>
> > a) the message should say that this is the *remaining* time to left to wait.
> >
> > b) it might be convenient to know from the log what was the original
> > min_apply_delay value in the 1st place.
> >
> > For example, the logs might look something like this:
> >
> > DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> > 160000 ms. Remaining wait time: 159972 ms
> > DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> > 160000 ms. Remaining wait time: 142828 ms
> > DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> > 160000 ms. Remaining wait time: 129994 ms
> > DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> > 160000 ms. Remaining wait time: 110001 ms
> > ...
> >
>
> +1
> This will also help when min_apply_delay is set to a new value in
> between the current wait. Lets say, I started with min_apply_delay=5
> min, when the worker was half way through this, I changed
> min_apply_delay to 3 min or say 10min, I see the impact of that change
> i.e. new wait-time is adjusted, but log becomes confusing. So, please
> keep this scenario as well in mind while improving logging.
>
when we send-feedback during apply-delay after every
wal_receiver_status_interval , the log comes as:
023-01-19 17:12:56.000 IST [404795] DEBUG: sending feedback (force 1)
to recv 0/1570840, write 0/1570840, flush 0/1570840
Shall we have some info here to indicate that it is sent while waiting
for apply_delay to distinguish it from other such send-feedback logs?
It will
make apply_delay flow clear in logs.
thanks
Shveta
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-20 18:36 Takamichi Osumi (Fujitsu) <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 1 reply; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-20 18:36 UTC (permalink / raw)
To: 'Peter Smith' <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Friday, January 20, 2023 3:56 PM Peter Smith <[email protected]> wrote:
> Hi Osumi-san, here are my review comments for the latest patch v17-0001.
Thanks for your review !
> ======
> Commit Message
>
> 1.
> Prohibit the combination of this feature and parallel streaming mode.
>
> SUGGESTION (using the same wording as in the code comments) The
> combination of parallel streaming mode and min_apply_delay is not allowed.
Okay. Fixed.
> ======
> doc/src/sgml/ref/create_subscription.sgml
>
> 2.
> + <para>
> + By default, the subscriber applies changes as soon as possible.
> This
> + parameter allows the user to delay the application of changes by a
> + specified amount of time. If the value is specified without units, it
> + is taken as milliseconds. The default is zero (no delay).
> + </para>
>
> Looking at this again, it seemed a bit strange to repeat "specified"
> twice in 2 sentences. Maybe change one of them.
>
> I’ve also suggested using the word "interval" because I don’t think docs yet
> mentioned anywhere (except in the example) that using intervals is possible.
>
> SUGGESTION (for the 2nd sentence)
> This parameter allows the user to delay the application of changes by a given
> time interval.
Adopted.
> ~~~
>
> 3.
> + <para>
> + Any delay occurs only on WAL records for transaction begins after
> all
> + initial table synchronization has finished. The delay is calculated
> + between the WAL timestamp as written on the publisher and the
> current
> + time on the subscriber. Any overhead of time spent in
> logical decoding
> + and in transferring the transaction may reduce the actual wait time.
> + It is also possible that the overhead already execeeds the
> requested
> + <literal>min_apply_delay</literal> value, in which case no
> additional
> + wait is necessary. If the system clocks on publisher and subscriber
> + are not synchronized, this may lead to apply changes earlier than
> + expected, but this is not a major issue because this parameter is
> + typically much larger than the time deviations between servers.
> Note
> + that if this parameter is set to a long delay, the replication will
> + stop if the replication slot falls behind the current LSN
> by more than
> + <link
> linkend="guc-max-slot-wal-keep-size"><literal>max_slot_wal_keep_size</
> literal></link>.
> + </para>
>
> 3a.
> Typo "execeeds" (I think Vignesh reported this already)
Fixed.
> ~
>
> 3b.
> SUGGESTION (for the 2nd sentence)
> BEFORE
> The delay is calculated between the WAL timestamp...
> AFTER
> The delay is calculated as the difference between the WAL timestamp...
Fixed.
> ~~~
>
> 4.
> + <warning>
> + <para>
> + Delaying the replication can mean there is a much longer
> time between making
> + a change on the publisher, and that change being
> committed on the subscriber.
> + v
> + See <xref linkend="guc-synchronous-commit"/>.
> + </para>
> + </warning>
>
> IMO maybe there is a better way to express the 2nd sentence:
>
> BEFORE
> This can have a big impact on synchronous replication.
> AFTER
> This can impact the performance of synchronous replication.
Fixed.
> ======
> src/backend/commands/subscriptioncmds.c
>
> 5. parse_subscription_options
>
> @@ -324,6 +328,43 @@ parse_subscription_options(ParseState *pstate, List
> *stmt_options,
> opts->specified_opts |= SUBOPT_LSN;
> opts->lsn = lsn;
> }
> + else if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
> + strcmp(defel->defname, "min_apply_delay") == 0) {
> + char *val,
> + *tmp;
> + Interval *interval;
> + int64 ms;
>
> IMO 'delay_ms' (or similar) would be a friendlier variable name than just 'ms'
The variable name has been changed which is more clear to the feature.
> ~~~
>
> 6.
> @@ -404,6 +445,20 @@ parse_subscription_options(ParseState *pstate, List
> *stmt_options,
> "slot_name = NONE", "create_slot = false")));
> }
> }
> +
> + /*
> + * The combination of parallel streaming mode and min_apply_delay is
> + not
> + * allowed.
> + */
> + if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
> + opts->min_apply_delay > 0)
> + {
> + if (opts->streaming == LOGICALREP_STREAM_PARALLEL)
> ereport(ERROR,
> + errcode(ERRCODE_SYNTAX_ERROR), errmsg("%s and %s are mutually
> + exclusive options",
> + "min_apply_delay > 0", "streaming = parallel")); }
>
> This could be expressed as a single condition using &&, maybe also with the
> brackets eliminated. (Unless you feel the current code is more readable)
The current style is intentional. We feel the code is more readable.
> ~~~
>
> 7.
>
> + if (opts.min_apply_delay > 0)
> + if ((IsSet(opts.specified_opts, SUBOPT_STREAMING) && opts.streaming
> == LOGICALREP_STREAM_PARALLEL) ||
> + (!IsSet(opts.specified_opts, SUBOPT_STREAMING) && sub->stream ==
> LOGICALREP_STREAM_PARALLEL))
> + ereport(ERROR,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("cannot enable %s for subscription in %s mode",
> + "min_apply_delay", "streaming = parallel"));
>
> These nested ifs could instead be a single "if" with && condition.
> (Unless you feel the current code is more readable)
Same as #6.
> ======
> src/backend/replication/logical/worker.c
>
> 8. maybe_delay_apply
>
> + * Hence, it's not appropriate to apply a delay at the time.
> + */
> +static void
> +maybe_delay_apply(TimestampTz finish_ts)
>
> That last sentence "Hence,... delay at the time" does not sound correct. Is there
> a typo or missing words here?
>
> Maybe it meant to say "... at the STREAM START time."?
Yes. Fixed.
> ~~~
>
> 9.
> + /* This might change wal_receiver_status_interval */ if
> + (ConfigReloadPending) { ConfigReloadPending = false;
> + ProcessConfigFile(PGC_SIGHUP); }
>
> I was unsure why did you make a special mention of
> 'wal_receiver_status_interval' here. I mean, Aren't there also other GUCs that
> might change and affect something here so was there some special reason only
> this one was mentioned?
This should be similar to the recoveryApplyDelay for physical replication.
It mentions the GUC used in the same function.
> ======
> src/test/subscription/t/032_apply_delay.pl
>
> 10.
> +
> +# Compare inserted time on the publisher with applied time on the
> +subscriber to # confirm the latter is applied after expected time.
> +sub check_apply_delay_time
>
> Maybe the comment could also mention that the time is automatically stored in
> the table column 'c'.
Added.
> ~~~
>
> 11.
> +# Confirm the suspended record doesn't get applied expectedly by the
> +ALTER # DISABLE command.
> +$result = $node_subscriber->safe_psql('postgres',
> + "SELECT count(a) FROM test_tab WHERE a = 0;"); is($result, qq(0),
> +"check if the delayed transaction doesn't get
> applied expectedly");
>
> The use of "doesn't get applied expectedly" (in 2 places here) seemed strange.
> Maybe it's better to say like
>
> SUGGESTION
> # Confirm disabling the subscription by ALTER DISABLE did not cause the
> delayed transaction to be applied.
> $result = $node_subscriber->safe_psql('postgres',
> "SELECT count(a) FROM test_tab WHERE a = 0;"); is($result, qq(0), "check
> the delayed transaction was not applied");
Fixed.
Kindly have a look at the patch v18.
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v18-0001-Time-delayed-logical-replication-subscriber.patch (81.8K, ../../TYCPR01MB8373BED9E390C4839AF56685EDC59@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v18-0001-Time-delayed-logical-replication-subscriber.patch)
download | inline diff:
From 737372d78eef4d2aa7ef8407ab0b6c8135fe55e1 Mon Sep 17 00:00:00 2001
From: Takamichi Osumi <[email protected]>
Date: Fri, 20 Jan 2023 17:50:47 +0000
Subject: [PATCH v18] Time-delayed logical replication subscriber
Similar to physical replication, a time-delayed copy of the data for
logical replication is useful for some scenarios (particularly to fix
errors that might cause data loss).
This patch implements a new subscription parameter called 'min_apply_delay'.
If the subscription sets min_apply_delay parameter, the logical
replication worker will delay the transaction commit for min_apply_delay
milliseconds.
The delay is calculated between the WAL time stamp and the current time
on the subscriber.
The delay occurs only on WAL records for transaction begins. The main
reason is to avoid keeping a transaction open for a long time. Regular
and prepared transactions are covered. Streamed transactions are also
covered.
The combination of parallel streaming mode and min_apply_delay is not
allowed. The subscriber in the parallel streaming mode applies each
stream on arrival without the time of commit/prepare. So, the subscriber
needs to depend on the arrival time of the stream in this case. Therefore,
if we apply the time-delayed feature for such transactions, then there is
a possibility where some unnecessary delay will be added on the subscriber
by network communication break between two nodes or other heavy work load
on the publisher. On the other hand, applying the delay at the end of
transaction with parallel apply also can cause issues of used resource
bloat and locks kept in open for a long time. Thus, those feature can't
be work together.
Author: Euler Taveira
Discussion: https://postgr.es/m/CAB-JLwYOYwL=XTyAXKiH5CtM_Vm8KjKh7aaitCKvmCh4rzr5pQ@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/config.sgml | 13 ++
doc/src/sgml/logical-replication.sgml | 7 +
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 57 +++++-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 7 +-
src/backend/commands/subscriptioncmds.c | 125 +++++++++++-
.../replication/logical/applyparallelworker.c | 3 +-
src/backend/replication/logical/worker.c | 169 ++++++++++++++--
src/bin/pg_dump/pg_dump.c | 15 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 9 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 3 +
src/include/datatype/timestamp.h | 2 +
src/include/replication/worker_internal.h | 2 +-
src/test/regress/expected/subscription.out | 181 +++++++++++-------
src/test/regress/sql/subscription.sql | 24 +++
src/test/subscription/meson.build | 1 +
src/test/subscription/t/032_apply_delay.pl | 175 +++++++++++++++++
21 files changed, 708 insertions(+), 105 deletions(-)
create mode 100644 src/test/subscription/t/032_apply_delay.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index c1e4048054..bf3c05241c 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7873,6 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subminapplydelay</structfield> <type>int8</type>
+ </para>
+ <para>
+ The length of time (ms) to delay the application of changes.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subname</structfield> <type>name</type>
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 89d53f2a64..13dd422c25 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4753,6 +4753,19 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
the <filename>postgresql.conf</filename> file or on the server
command line.
</para>
+ <para>
+ For time-delayed logical replication (i.e. when the subscription is
+ created with parameter min_apply_delay > 0), the apply worker sends a
+ Standby Status Update message to the publisher with a period of
+ <literal>wal_receiver_status_interval</literal>. Make sure to set
+ <literal>wal_receiver_status_interval</literal> less than the
+ <literal>wal_sender_timeout</literal> on the publisher, otherwise, the
+ walsender will repeatedly terminate due to the timeout errors. If
+ <literal>wal_receiver_status_interval</literal> is set to zero, the apply
+ worker doesn't send any feedback messages during the subscriber's
+ <literal>min_apply_delay</literal> period. See
+ <xref linkend="sql-createsubscription"/> for details.
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 1bd5660c87..863af11a47 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -247,6 +247,13 @@
target table.
</para>
+ <para>
+ The subscriber replication can be instructed to lag behind the publisher
+ side changes by specifying the <literal>min_apply_delay</literal>
+ subscription parameter. See <xref linkend="sql-createsubscription"/> for
+ details.
+ </para>
+
<sect2 id="logical-replication-subscription-slot">
<title>Replication Slot Management</title>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index ad93553a1d..1c6e9dd2d1 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -213,8 +213,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
<literal>binary</literal>, <literal>streaming</literal>,
- <literal>disable_on_error</literal>, and
- <literal>origin</literal>.
+ <literal>disable_on_error</literal>,
+ <literal>origin</literal>, and
+ <literal>min_apply_delay</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index eba72c6af6..76ee9c0b3d 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -349,7 +349,45 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
- </variablelist></para>
+
+ <varlistentry>
+ <term><literal>min_apply_delay</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ By default, the subscriber applies changes as soon as possible. This
+ parameter allows the user to delay the application of changes by a
+ given time interval. If the value is specified without units, it is
+ taken as milliseconds. The default is zero (no delay).
+ </para>
+ <para>
+ Any delay occurs only on WAL records for transaction begins after all
+ initial table synchronization has finished. The delay is calculated
+ as the difference between the WAL timestamp as written on the
+ publisher and the current time on the subscriber. Any overhead of
+ time spent in logical decoding and in transferring the transaction
+ may reduce the actual wait time. It is also possible that the overhead
+ already exceeds the requested <literal>min_apply_delay</literal> value,
+ in which case no additional wait is necessary. If the system clocks
+ on publisher and subscriber are not synchronized, this may lead to
+ apply changes earlier than expected, but this is not a major issue
+ because this parameter is typically much larger than the time
+ deviations between servers. Note that if this parameter is set to a
+ long delay, the replication will stop if the replication slot falls
+ behind the current LSN by more than
+ <link linkend="guc-max-slot-wal-keep-size"><literal>max_slot_wal_keep_size</literal></link>.
+ </para>
+ <warning>
+ <para>
+ Delaying the replication can mean there is a much longer time between making
+ a change on the publisher, and that change being committed on the subscriber.
+ This can impact the performance of synchronous replication.
+ See <xref linkend="guc-synchronous-commit"/>.
+ </para>
+ </warning>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
@@ -413,6 +451,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
published with different column lists are not supported.
</para>
+ <para>
+ A non-zero <literal>min_apply_delay</literal> parameter is not allowed when streaming
+ in parallel mode.
+ </para>
+
<para>
We allow non-existent publications to be specified so that users can add
those later. This means
@@ -472,6 +515,18 @@ CREATE SUBSCRIPTION mysub
PUBLICATION insert_only
WITH (enabled = false);
</programlisting></para>
+
+ <para>
+ Create a subscription to a remote server that replicates tables in
+ the <literal>mypub</literal> publication and starts replicating immediately
+ on commit. Pre-existing data is not copied. The application of changes is
+ delayed by 4 hours.
+<programlisting>
+CREATE SUBSCRIPTION mysub
+ CONNECTION 'host=192.0.2.4 port=5432 user=foo dbname=foodb'
+ PUBLICATION mypub
+ WITH (copy_data = false, min_apply_delay = '4h');
+</programlisting></para>
</refsect1>
<refsect1>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index a56ae311c3..c767cc1c3a 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -64,6 +64,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->oid = subid;
sub->dbid = subform->subdbid;
sub->skiplsn = subform->subskiplsn;
+ sub->minapplydelay = subform->subminapplydelay;
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 8608e3fa5b..317c2010cb 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1299,9 +1299,10 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
- subbinary, substream, subtwophasestate, subdisableonerr,
- subslotname, subsynccommit, subpublications, suborigin)
+GRANT SELECT (oid, subdbid, subskiplsn, subminapplydelay, subname, subowner,
+ subenabled, subbinary, substream, subtwophasestate,
+ subdisableonerr, subslotname, subsynccommit, subpublications,
+ suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index baff00dd74..68f6f76102 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -66,6 +66,7 @@
#define SUBOPT_DISABLE_ON_ERR 0x00000400
#define SUBOPT_LSN 0x00000800
#define SUBOPT_ORIGIN 0x00001000
+#define SUBOPT_MIN_APPLY_DELAY 0x00002000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -90,6 +91,7 @@ typedef struct SubOpts
bool disableonerr;
char *origin;
XLogRecPtr lsn;
+ int min_apply_delay;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -100,7 +102,7 @@ static void check_publications_origin(WalReceiverConn *wrconn,
static void check_duplicates_in_publist(List *publist, Datum *datums);
static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, const char *subname);
static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err);
-
+static int defGetMinApplyDelay(DefElem *def);
/*
* Common option parsing function for CREATE and ALTER SUBSCRIPTION commands.
@@ -146,6 +148,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->disableonerr = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY))
+ opts->min_apply_delay = 0;
/* Parse options */
foreach(lc, stmt_options)
@@ -324,6 +328,16 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ strcmp(defel->defname, "min_apply_delay") == 0)
+ {
+ int min_apply_delay;
+
+ opts->specified_opts |= SUBOPT_MIN_APPLY_DELAY;
+ min_apply_delay = defGetMinApplyDelay(defel);
+
+ opts->min_apply_delay = min_apply_delay;
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -404,6 +418,29 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * The combination of parallel streaming mode and min_apply_delay is not
+ * allowed. The subscriber in the parallel streaming mode applies each
+ * stream on arrival without the time of commit/prepare. So, the
+ * subscriber needs to depend on the arrival time of the stream in this
+ * case. Therefore, if we apply the time-delayed feature for such
+ * transactions, then there is a possibility where some unnecessary delay
+ * will be added on the subscriber by network communication break between
+ * nodes or other heavy work load on the publisher. On the other hand,
+ * applying the delay at the end of transaction with parallel apply also
+ * can cause issues of used resource bloat and locks kept in open for a
+ * long time. Thus, those feature can't be work together.
+ */
+ if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ opts->min_apply_delay > 0)
+ {
+ if (opts->streaming == LOGICALREP_STREAM_PARALLEL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s and %s are mutually exclusive options",
+ "min_apply_delay > 0", "streaming = parallel"));
+ }
}
/*
@@ -560,7 +597,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
- SUBOPT_DISABLE_ON_ERR | SUBOPT_ORIGIN);
+ SUBOPT_DISABLE_ON_ERR | SUBOPT_ORIGIN |
+ SUBOPT_MIN_APPLY_DELAY);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -625,6 +663,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_oid - 1] = ObjectIdGetDatum(subid);
values[Anum_pg_subscription_subdbid - 1] = ObjectIdGetDatum(MyDatabaseId);
values[Anum_pg_subscription_subskiplsn - 1] = LSNGetDatum(InvalidXLogRecPtr);
+ values[Anum_pg_subscription_subminapplydelay - 1] = Int64GetDatum(opts.min_apply_delay);
values[Anum_pg_subscription_subname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
@@ -1054,7 +1093,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
- SUBOPT_ORIGIN);
+ SUBOPT_ORIGIN | SUBOPT_MIN_APPLY_DELAY);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1098,6 +1137,18 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
if (IsSet(opts.specified_opts, SUBOPT_STREAMING))
{
+ /*
+ * The combination of parallel streaming mode and
+ * min_apply_delay is not allowed.
+ */
+ if (opts.streaming == LOGICALREP_STREAM_PARALLEL)
+ if ((IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) && opts.min_apply_delay > 0) ||
+ (!IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) && sub->minapplydelay > 0))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s mode for subscription with %s",
+ "streaming = parallel", "min_apply_delay"));
+
values[Anum_pg_subscription_substream - 1] =
CharGetDatum(opts.streaming);
replaces[Anum_pg_subscription_substream - 1] = true;
@@ -1111,6 +1162,25 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
= true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY))
+ {
+ /*
+ * The combination of parallel streaming mode and
+ * min_apply_delay is not allowed.
+ */
+ if (opts.min_apply_delay > 0)
+ if ((IsSet(opts.specified_opts, SUBOPT_STREAMING) && opts.streaming == LOGICALREP_STREAM_PARALLEL) ||
+ (!IsSet(opts.specified_opts, SUBOPT_STREAMING) && sub->stream == LOGICALREP_STREAM_PARALLEL))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s for subscription in %s mode",
+ "min_apply_delay", "streaming = parallel"));
+
+ values[Anum_pg_subscription_subminapplydelay - 1] =
+ Int64GetDatum(opts.min_apply_delay);
+ replaces[Anum_pg_subscription_subminapplydelay - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
@@ -2185,3 +2255,52 @@ defGetStreamingMode(DefElem *def)
def->defname)));
return LOGICALREP_STREAM_OFF; /* keep compiler quiet */
}
+
+
+/*
+ * Extract the min_apply_delay mode value from a DefElem. This is very similar
+ * to PGC_INT case of parse_and_validate_value(), because min_apply_delay
+ * accepts the same string as recovery_min_apply_delay,
+ */
+int
+defGetMinApplyDelay(DefElem *def)
+{
+ char *value;
+ int result;
+ const char *hintmsg;
+
+ /*
+ * Raise an ERROR if no parameter value given
+ */
+ if (def->arg == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s requires an integer value",
+ def->defname)));
+
+ value = defGetString(def);
+
+ /*
+ * Parse given string as parameter which has millisecond unit
+ */
+ if (!parse_int(value, &result, GUC_UNIT_MS, &hintmsg))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid value for parameter \"%s\": \"%s\"",
+ "min_apply_delay", value),
+ hintmsg ? errhint("%s", _(hintmsg)) : 0));
+
+ /*
+ * Check lower bound. parse_int() has been already confirmed that result
+ * is equal to or smaller than INT_MAX.
+ */
+ if (result < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d ms is outside the valid range for parameter \"%s\" (%d .. %d)",
+ result,
+ "min_apply_delay",
+ 0, INT_MAX)));
+
+ return result;
+}
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index 3579e704fe..7302bce7a0 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -704,7 +704,8 @@ pa_process_spooled_messages_if_required(void)
{
apply_spooled_messages(&MyParallelShared->fileset,
MyParallelShared->xid,
- InvalidXLogRecPtr);
+ InvalidXLogRecPtr,
+ 0);
pa_set_fileset_state(MyParallelShared, FS_EMPTY);
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a0084c7ef6..31719db030 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -318,6 +318,17 @@ static List *on_commit_wakeup_workers_subids = NIL;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/*
+ * In order to avoid walsender timeout for time-delayed replication the worker
+ * process keeps sending feedback messages during the delay period.
+ * Meanwhile, the feature delays the apply before starting the
+ * transaction and thus we don't write WALs for the suspended changes during
+ * the wait. When the worker process sends a feedback message
+ * during the delay, we should not make positions of the flushed and apply LSN
+ * overwritten by the last received latest LSN. See send_feedback() for details.
+ */
+static XLogRecPtr last_received = InvalidXLogRecPtr;
+
/* fields valid only when processing streamed transaction */
static bool in_streamed_transaction = false;
@@ -388,10 +399,13 @@ static void stream_write_change(char action, StringInfo s);
static void stream_open_and_write_change(TransactionId xid, char action, StringInfo s);
static void stream_close_file(void);
-static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
+static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply,
+ bool in_delayed_apply);
static void DisableSubscriptionAndExit(void);
+static void maybe_delay_apply(TransactionId xid, TimestampTz finish_ts);
+
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
@@ -998,6 +1012,108 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
ExecStoreVirtualTuple(slot);
}
+/*
+ * When min_apply_delay parameter is set on the subscriber, we wait long enough
+ * to make sure a transaction is applied at least that interval behind the
+ * publisher.
+ *
+ * While the physical replication applies the delay at commit time, this
+ * feature applies the delay for the next transaction but before starting the
+ * transaction. This is mainly because keeping a transaction that conducted
+ * write operations open for a long time results in some issues such as bloat
+ * and locks.
+ *
+ * The min_apply_delay parameter will take effect only after all tables are in
+ * READY state.
+ *
+ * xid is the transaction id where we apply the delay.
+ *
+ * finish_ts is the commit/prepare time of both regular (non-streamed) and
+ * streamed transactions. Unlike the regular (non-streamed) cases, the delay
+ * is applied in a STREAM COMMIT/STREAM PREPARE message for streamed
+ * transactions. The STREAM START message does not contain a commit/prepare
+ * time (it will be available when the in-progress transaction finishes).
+ * Hence, it's not appropriate to apply a delay at the STREAM START time.
+ */
+static void
+maybe_delay_apply(TransactionId xid, TimestampTz finish_ts)
+{
+ Assert(finish_ts > 0);
+
+ /* Nothing to do if no delay set */
+ if (!MySubscription->minapplydelay)
+ return;
+
+ /*
+ * The min_apply_delay parameter is ignored until all tablesync workers
+ * have reached READY state. This is because if we allowed the delay
+ * during the catchup phase, then once we reached the limit of tablesync
+ * workers it would impose a delay for each subsequent worker. That would
+ * cause initial table synchronization completion to take a long time.
+ */
+ if (!AllTablesyncsReady())
+ return;
+
+ /* Apply the delay by the latch mechanism */
+ while (true)
+ {
+ long diffms;
+
+ ResetLatch(MyLatch);
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* This might change wal_receiver_status_interval */
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /*
+ * Before calculating the time duration, reload the catalog if needed.
+ */
+ if (!in_remote_transaction && !in_streamed_transaction)
+ {
+ AcceptInvalidationMessages();
+ maybe_reread_subscription();
+ }
+
+ diffms = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ TimestampTzPlusMilliseconds(finish_ts, MySubscription->minapplydelay));
+
+ /*
+ * Exit without arming the latch if it's already past time to apply
+ * this transaction.
+ */
+ if (diffms <= 0)
+ break;
+
+ elog(DEBUG2, "time-delayed replication for txid %u, min_apply_delay = %lld ms, Remaining wait time: %ld ms",
+ xid, (long long) MySubscription->minapplydelay, diffms);
+
+ /*
+ * Call send_feedback() to prevent the publisher from exiting by
+ * timeout during the delay, when wal_receiver_status_interval is
+ * available.
+ */
+ if (wal_receiver_status_interval > 0
+ && diffms > wal_receiver_status_interval * 1000)
+ {
+ WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ (long) wal_receiver_status_interval * 1000,
+ WAIT_EVENT_RECOVERY_APPLY_DELAY);
+ send_feedback(last_received, true, false, true);
+ }
+ else
+ WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ diffms,
+ WAIT_EVENT_RECOVERY_APPLY_DELAY);
+ }
+}
+
/*
* Handle BEGIN message.
*/
@@ -1012,6 +1128,9 @@ apply_handle_begin(StringInfo s)
logicalrep_read_begin(s, &begin_data);
set_apply_error_context_xact(begin_data.xid, begin_data.final_lsn);
+ /* Should we delay the current transaction? */
+ maybe_delay_apply(begin_data.xid, begin_data.committime);
+
remote_final_lsn = begin_data.final_lsn;
maybe_start_skipping_changes(begin_data.final_lsn);
@@ -1069,6 +1188,9 @@ apply_handle_begin_prepare(StringInfo s)
logicalrep_read_begin_prepare(s, &begin_data);
set_apply_error_context_xact(begin_data.xid, begin_data.prepare_lsn);
+ /* Should we delay the current prepared transaction? */
+ maybe_delay_apply(begin_data.xid, begin_data.prepare_time);
+
remote_final_lsn = begin_data.prepare_lsn;
maybe_start_skipping_changes(begin_data.prepare_lsn);
@@ -1316,7 +1438,8 @@ apply_handle_stream_prepare(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset,
- prepare_data.xid, prepare_data.prepare_lsn);
+ prepare_data.xid, prepare_data.prepare_lsn,
+ prepare_data.prepare_time);
/* Mark the transaction as prepared. */
apply_handle_prepare_internal(&prepare_data);
@@ -2010,10 +2133,13 @@ ensure_last_message(FileSet *stream_fileset, TransactionId xid, int fileno,
/*
* Common spoolfile processing.
+ *
+ * The commit/prepare time for streaming transaction is required to achieve
+ * time-delayed replication.
*/
void
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn, TimestampTz finish_ts)
{
StringInfoData s2;
int nchanges;
@@ -2024,6 +2150,10 @@ apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
int fileno;
off_t offset;
+ /* Should we delay the current transaction? */
+ if (finish_ts)
+ maybe_delay_apply(xid, finish_ts);
+
if (!am_parallel_apply_worker())
maybe_start_skipping_changes(lsn);
@@ -2173,7 +2303,7 @@ apply_handle_stream_commit(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset, xid,
- commit_data.commit_lsn);
+ commit_data.commit_lsn, commit_data.committime);
apply_handle_commit_internal(&commit_data);
@@ -3446,7 +3576,7 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
* Apply main loop.
*/
static void
-LogicalRepApplyLoop(XLogRecPtr last_received)
+LogicalRepApplyLoop(void)
{
TimestampTz last_recv_timestamp = GetCurrentTimestamp();
bool ping_sent = false;
@@ -3567,7 +3697,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
if (last_received < end_lsn)
last_received = end_lsn;
- send_feedback(last_received, reply_requested, false);
+ send_feedback(last_received, reply_requested, false, false);
UpdateWorkerStats(last_received, timestamp, true);
}
/* other message types are purposefully ignored */
@@ -3580,7 +3710,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/* confirm all writes so far */
- send_feedback(last_received, false, false);
+ send_feedback(last_received, false, false, false);
if (!in_remote_transaction && !in_streamed_transaction)
{
@@ -3677,7 +3807,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
}
- send_feedback(last_received, requestReply, requestReply);
+ send_feedback(last_received, requestReply, requestReply, false);
/*
* Force reporting to ensure long idle periods don't lead to
@@ -3707,7 +3837,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
* to send a response to avoid timeouts.
*/
static void
-send_feedback(XLogRecPtr recvpos, bool force, bool requestReply)
+send_feedback(XLogRecPtr recvpos, bool force, bool requestReply, bool in_delayed_apply)
{
static StringInfo reply_message = NULL;
static TimestampTz send_time = 0;
@@ -3737,8 +3867,15 @@ send_feedback(XLogRecPtr recvpos, bool force, bool requestReply)
/*
* No outstanding transactions to flush, we can report the latest received
* position. This is important for synchronous replication.
+ *
+ * If the subscriber side apply is delayed (because of time-delayed
+ * replication) then do not tell the publisher that the received latest
+ * LSN is already applied and flushed, otherwise, it leads to the
+ * publisher side making a wrong assumption of logical replication
+ * progress. Instead, we just send a feedback message to avoid a publisher
+ * timeout during the delay.
*/
- if (!have_pending_txes)
+ if (!have_pending_txes && !in_delayed_apply)
flushpos = writepos = recvpos;
if (writepos < last_writepos)
@@ -3775,11 +3912,12 @@ send_feedback(XLogRecPtr recvpos, bool force, bool requestReply)
pq_sendint64(reply_message, now); /* sendTime */
pq_sendbyte(reply_message, requestReply); /* replyRequested */
- elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write %X/%X, flush %X/%X",
+ elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write %X/%X, flush %X/%X in-delayed: %d",
force,
LSN_FORMAT_ARGS(recvpos),
LSN_FORMAT_ARGS(writepos),
- LSN_FORMAT_ARGS(flushpos));
+ LSN_FORMAT_ARGS(flushpos),
+ in_delayed_apply);
walrcv_send(LogRepWorkerWalRcvConn,
reply_message->data, reply_message->len);
@@ -4354,11 +4492,11 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname)
* of system resource error and are not repeatable.
*/
static void
-start_apply(XLogRecPtr origin_startpos)
+start_apply(void)
{
PG_TRY();
{
- LogicalRepApplyLoop(origin_startpos);
+ LogicalRepApplyLoop();
}
PG_CATCH();
{
@@ -4645,7 +4783,8 @@ ApplyWorkerMain(Datum main_arg)
}
/* Run the main loop. */
- start_apply(origin_startpos);
+ last_received = origin_startpos;
+ start_apply();
proc_exit(0);
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 527c7651ab..c0f69cb43b 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4494,6 +4494,7 @@ getSubscriptions(Archive *fout)
int i_subsynccommit;
int i_subpublications;
int i_subbinary;
+ int i_subminapplydelay;
int i,
ntups;
@@ -4546,9 +4547,13 @@ getSubscriptions(Archive *fout)
LOGICALREP_TWOPHASE_STATE_DISABLED);
if (fout->remoteVersion >= 160000)
- appendPQExpBufferStr(query, " s.suborigin\n");
+ appendPQExpBufferStr(query,
+ " s.suborigin,\n"
+ " s.subminapplydelay\n");
else
- appendPQExpBuffer(query, " '%s' AS suborigin\n", LOGICALREP_ORIGIN_ANY);
+ appendPQExpBuffer(query, " '%s' AS suborigin,\n"
+ " 0 AS subminapplydelay\n",
+ LOGICALREP_ORIGIN_ANY);
appendPQExpBufferStr(query,
"FROM pg_subscription s\n"
@@ -4576,6 +4581,7 @@ getSubscriptions(Archive *fout)
i_subtwophasestate = PQfnumber(res, "subtwophasestate");
i_subdisableonerr = PQfnumber(res, "subdisableonerr");
i_suborigin = PQfnumber(res, "suborigin");
+ i_subminapplydelay = PQfnumber(res, "subminapplydelay");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4606,6 +4612,8 @@ getSubscriptions(Archive *fout)
subinfo[i].subdisableonerr =
pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
subinfo[i].suborigin = pg_strdup(PQgetvalue(res, i, i_suborigin));
+ subinfo[i].subminapplydelay =
+ strtoi64(PQgetvalue(res, i, i_subminapplydelay), NULL, 10);
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -4687,6 +4695,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
+ if (subinfo->subminapplydelay > 0)
+ appendPQExpBuffer(query, ", min_apply_delay = '" INT64_FORMAT " ms'", subinfo->subminapplydelay);
+
appendPQExpBufferStr(query, ");\n");
if (subinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index e7cbd8d7ed..e2525f70ab 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -661,6 +661,7 @@ typedef struct _SubscriptionInfo
char *subdisableonerr;
char *suborigin;
char *subsynccommit;
+ int64 subminapplydelay;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index c8a0bb7b3a..8a27063bed 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6472,7 +6472,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false, false, false, false};
+ false, false, false, false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6527,10 +6527,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Two-phase commit"),
gettext_noop("Disable on error"));
+ /* Origin and min_apply_delay are only supported in v16 and higher */
if (pset.sversion >= 160000)
appendPQExpBuffer(&buf,
- ", suborigin AS \"%s\"\n",
- gettext_noop("Origin"));
+ ", suborigin AS \"%s\"\n"
+ ", subminapplydelay AS \"%s\"\n",
+ gettext_noop("Origin"),
+ gettext_noop("Min apply delay (ms)"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 5e1882eaea..e8b9a43a47 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1925,7 +1925,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "origin", "slot_name",
+ COMPLETE_WITH("binary", "disable_on_error", "min_apply_delay", "origin", "slot_name",
"streaming", "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
@@ -3268,7 +3268,7 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "origin", "slot_name",
+ "disable_on_error", "enabled", "min_apply_delay", "origin", "slot_name",
"streaming", "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index b0f2a1705d..e06f35c037 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -70,6 +70,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
XLogRecPtr subskiplsn; /* All changes finished at this LSN are
* skipped */
+ int64 subminapplydelay; /* Replication apply delay (ms) */
+
NameData subname; /* Name of the subscription */
Oid subowner BKI_LOOKUP(pg_authid); /* Owner of the subscription */
@@ -120,6 +122,7 @@ typedef struct Subscription
* in */
XLogRecPtr skiplsn; /* All changes finished at this LSN are
* skipped */
+ int64 minapplydelay; /* Replication apply delay (ms) */
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
diff --git a/src/include/datatype/timestamp.h b/src/include/datatype/timestamp.h
index 21a37e21e9..8b368af299 100644
--- a/src/include/datatype/timestamp.h
+++ b/src/include/datatype/timestamp.h
@@ -127,6 +127,8 @@ struct pg_itm_in
#define SECS_PER_MINUTE 60
#define MINS_PER_HOUR 60
+#define MSECS_PER_DAY INT64CONST(86400000)
+
#define USECS_PER_DAY INT64CONST(86400000000)
#define USECS_PER_HOUR INT64CONST(3600000000)
#define USECS_PER_MINUTE INT64CONST(60000000)
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index dc87a4edd1..3dc09d1a4c 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -255,7 +255,7 @@ extern void stream_stop_internal(TransactionId xid);
/* Common streaming function to apply all the spooled messages */
extern void apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn);
+ XLogRecPtr lsn, TimestampTz finish_ts);
extern void apply_dispatch(StringInfo s);
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 4e5cb0d3a9..89ca5505f8 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -114,18 +114,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -135,10 +135,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -155,10 +155,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -167,10 +167,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -202,10 +202,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -239,19 +239,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -263,27 +263,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -298,10 +298,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -316,10 +316,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -355,10 +355,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -367,10 +367,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -380,10 +380,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,20 +396,57 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail -- min_apply_delay must be a non-negative integer
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = foo);
+ERROR: invalid value for parameter "min_apply_delay": "foo"
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = -1);
+ERROR: -1 ms is outside the valid range for parameter "min_apply_delay" (0 .. 2147483647)
+-- fail - utilizing streaming = parallel with time-delayed replication is not supported.
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = parallel, min_apply_delay = 123);
+ERROR: min_apply_delay > 0 and streaming = parallel are mutually exclusive options
+-- success -- min_apply_delay value without unit is taken as milliseconds
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = 123);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 123 | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+-- success -- min_apply_delay value with unit is converted into ms and stored as an integer
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = '1 d');
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 86400000 | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+-- fail - alter subscription with streaming = parallel should fail when time-delayed replication is set.
+ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
+ERROR: cannot enable streaming = parallel mode for subscription with min_apply_delay
+-- fail - alter subscription with min_apply_delay should fail when streaming = parallel is set.
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 0, streaming = parallel);
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 123);
+ERROR: cannot enable min_apply_delay for subscription in streaming = parallel mode
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
RESET SESSION AUTHORIZATION;
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 5f27b7d776..d6b893a3d0 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -279,6 +279,30 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail -- min_apply_delay must be a non-negative integer
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = foo);
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = -1);
+
+-- fail - utilizing streaming = parallel with time-delayed replication is not supported.
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = parallel, min_apply_delay = 123);
+
+-- success -- min_apply_delay value without unit is taken as milliseconds
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = 123);
+\dRs+
+
+-- success -- min_apply_delay value with unit is converted into ms and stored as an integer
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = '1 d');
+\dRs+
+
+-- fail - alter subscription with streaming = parallel should fail when time-delayed replication is set.
+ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
+
+-- fail - alter subscription with min_apply_delay should fail when streaming = parallel is set.
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 0, streaming = parallel);
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 123);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index 3db0fdfd96..a186876eb4 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -38,6 +38,7 @@ tests += {
't/029_on_error.pl',
't/030_origin.pl',
't/031_column_list.pl',
+ 't/032_apply_delay.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/032_apply_delay.pl b/src/test/subscription/t/032_apply_delay.pl
new file mode 100644
index 0000000000..51dde83cec
--- /dev/null
+++ b/src/test/subscription/t/032_apply_delay.pl
@@ -0,0 +1,175 @@
+
+# Copyright (c) 2023, PostgreSQL Global Development Group
+
+# Test replication apply delay
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Confirm the time-delayed replication has been effective from the server log
+# message where the apply worker emits for applying delay. Moreover, verifies
+# that the current worker's delayed time is sufficiently bigger than the
+# expected value, in order to check any update of the min_apply_delay.
+sub check_apply_delay_log
+{
+ my ($node_subscriber, $offset, $expected) = @_;
+
+ my $log_location = $node_subscriber->wait_for_log(qr/time-delayed replication for txid (\d+), min_apply_delay = (\d+) ms, Remaining wait time: (\d+) ms/, $offset);
+
+ cmp_ok($log_location, '>', $offset,
+ "logfile contains triggered logical replication apply delay"
+ );
+
+ # Get the delay time from the server log
+ my $contents = slurp_file($node_subscriber->logfile, $offset);
+ $contents =~
+ qr/time-delayed replication for txid (\d+), min_apply_delay = (\d+) ms, Remaining wait time: (\d+) ms/
+ or die "could not get the apply worker wait time";
+ my $logged_delay = $3;
+
+ # Is it larger than expected?
+ cmp_ok($logged_delay, '>', $expected,
+ "The apply worker wait time has expected duration"
+ );
+}
+
+# Compare inserted time on the publisher with applied time on the subscriber to
+# confirm the latter is applied after expected time. The time is automatically
+# generated and stored in the table column 'c'.
+sub check_apply_delay_time
+{
+ my ($node_publisher, $node_subscriber, $primary_key, $expected_diffs) = @_;
+
+ my $inserted_time_on_pub = $node_publisher->safe_psql('postgres', qq[
+ SELECT extract(epoch from c) FROM test_tab WHERE a = $primary_key;
+ ]);
+
+ my $inserted_time_on_sub = $node_subscriber->safe_psql('postgres', qq[
+ SELECT extract(epoch from c) FROM test_tab WHERE a = $primary_key;
+ ]);
+
+ cmp_ok($inserted_time_on_sub - $inserted_time_on_pub, '>', $expected_diffs,
+ "The tuple on the subscriber was modified later than the publisher");
+}
+
+# Create publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = debug2");
+$node_subscriber->start;
+
+# Setup structure on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar, c timestamptz (6) DEFAULT now())");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b text, c timestamptz (6) DEFAULT now())"
+);
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+# The column c must not be published because we want to compare the time
+# difference.
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE test_tab (a, b)");
+
+my $appname = 'tap_sub';
+
+# Create a subscription that applies the trasaction after 50 milliseconds delay
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (copy_data = off, min_apply_delay = '50ms', streaming = 'on')"
+);
+
+# New row to trigger apply delay
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (2, 'bar')");
+
+$node_publisher->wait_for_catchup($appname);
+
+my $result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM test_tab");
+is($result, qq(2|1|2), 'check if the new rows were applied to subscriber');
+
+# Note that we cannot call check_apply_delay_log() here because there is a
+# possibility that the delay is skipped. The event happens when the WAL
+# replication between publisher and subscriber is delayed due to a mechanical
+# problem. The log output will be checked later - substantial delay-time case.
+
+# Verify that the subscriber lags the publisher by at least 50 milliseconds
+check_apply_delay_time($node_publisher, $node_subscriber, '2', '0.05');
+
+# Setup for streaming case
+$node_publisher->append_conf('postgres.conf',
+ 'logical_decoding_mode = immediate');
+$node_publisher->reload;
+
+# Run a query to make sure that the reload has taken effect.
+$node_publisher->safe_psql('postgres', q{SELECT 1});
+
+# Test streamed transaction by insert
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5) s(i);");
+
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM test_tab");
+is($result, qq(5|1|5), 'check if the new rows were applied to subscriber');
+
+# Verify that the subscriber lags the publisher by at least 50 milliseconds
+check_apply_delay_time($node_publisher, $node_subscriber, '5', '0.05');
+
+# Test whether ALTER SUBSCRIPTION changes the delayed time of the apply worker
+# (1 day 5 minutes). Note that the extra 5 minute is to account for any
+# decoding/network overhead.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (min_apply_delay = 86700000)"
+);
+
+# Check log starting now for logical replication apply delay
+my $offset = -s $node_subscriber->logfile;
+
+# New row to trigger apply delay
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (0, 'foobar')");
+
+# Make sure the apply worker knows to wait for more than 1 day
+check_apply_delay_log($node_subscriber, $offset, "86400000");
+
+# Disable subscription and the worker should die immediately
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub DISABLE;"
+);
+
+# Wait until worker dies
+my $sub_query =
+ "SELECT count(1) = 0 FROM pg_stat_subscription WHERE subname = 'tap_sub' AND pid IS NOT NULL;";
+$node_subscriber->poll_query_until('postgres', $sub_query)
+ or die "Timed out while waiting for subscriber to die";
+
+# Confirm disabling the subscription by ALTER SUBSCRIPTION DISABLE did not cause
+# the delayed transaction to be applied.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(a) FROM test_tab WHERE a = 0;");
+is($result, qq(0), "check the delayed transaction was not applied");
+
+$node_subscriber->stop;
+$node_publisher->stop;
+
+done_testing();
--
2.30.0
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-20 18:41 Takamichi Osumi (Fujitsu) <[email protected]>
parent: shveta malik <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-20 18:41 UTC (permalink / raw)
To: 'shveta malik' <[email protected]>; Peter Smith <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
Hi,
On Friday, January 20, 2023 6:13 PM shveta malik <[email protected]> wrote:
> On Fri, Jan 20, 2023 at 2:23 PM shveta malik <[email protected]> wrote:
> >
> > On Fri, Jan 20, 2023 at 1:08 PM Peter Smith <[email protected]>
> wrote:
> >
> > > a) the message should say that this is the *remaining* time to left to wait.
> > >
> > > b) it might be convenient to know from the log what was the original
> > > min_apply_delay value in the 1st place.
> > >
> > > For example, the logs might look something like this:
> > >
> > > DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> > > 160000 ms. Remaining wait time: 159972 ms
> > > DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> > > 160000 ms. Remaining wait time: 142828 ms
> > > DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> > > 160000 ms. Remaining wait time: 129994 ms
> > > DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> > > 160000 ms. Remaining wait time: 110001 ms ...
> > >
> >
> > +1
> > This will also help when min_apply_delay is set to a new value in
> > between the current wait. Lets say, I started with min_apply_delay=5
> > min, when the worker was half way through this, I changed
> > min_apply_delay to 3 min or say 10min, I see the impact of that change
> > i.e. new wait-time is adjusted, but log becomes confusing. So, please
> > keep this scenario as well in mind while improving logging.
> >
>
>
> when we send-feedback during apply-delay after every
> wal_receiver_status_interval , the log comes as:
> 023-01-19 17:12:56.000 IST [404795] DEBUG: sending feedback (force 1) to
> recv 0/1570840, write 0/1570840, flush 0/1570840
>
> Shall we have some info here to indicate that it is sent while waiting for
> apply_delay to distinguish it from other such send-feedback logs?
> It will
> make apply_delay flow clear in logs.
This additional tip of log information has been added in the latest v18.
Kindly have a look at it in [1].
[1] - https://www.postgresql.org/message-id/TYCPR01MB8373BED9E390C4839AF56685EDC59%40TYCPR01MB8373.jpnprd0...
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-20 18:46 Takamichi Osumi (Fujitsu) <[email protected]>
parent: shveta malik <[email protected]>
1 sibling, 0 replies; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-20 18:46 UTC (permalink / raw)
To: 'shveta malik' <[email protected]>; Peter Smith <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Friday, January 20, 2023 5:54 PM shveta malik <[email protected]> wrote:
> On Fri, Jan 20, 2023 at 1:08 PM Peter Smith <[email protected]> wrote:
>
> > a) the message should say that this is the *remaining* time to left to wait.
> >
> > b) it might be convenient to know from the log what was the original
> > min_apply_delay value in the 1st place.
> >
> > For example, the logs might look something like this:
> >
> > DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> > 160000 ms. Remaining wait time: 159972 ms
> > DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> > 160000 ms. Remaining wait time: 142828 ms
> > DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> > 160000 ms. Remaining wait time: 129994 ms
> > DEBUG: time-delayed replication for txid 1234, min_apply_delay =
> > 160000 ms. Remaining wait time: 110001 ms ...
> >
>
> +1
> This will also help when min_apply_delay is set to a new value in between the
> current wait. Lets say, I started with min_apply_delay=5 min, when the worker
> was half way through this, I changed min_apply_delay to 3 min or say 10min, I
> see the impact of that change i.e. new wait-time is adjusted, but log becomes
> confusing. So, please keep this scenario as well in mind while improving
> logging.
Yes, now the change of min_apply_delay value can be detected
since I followed the format provided above. So, this scenario is also covered.
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-20 18:50 Takamichi Osumi (Fujitsu) <[email protected]>
parent: shveta malik <[email protected]>
1 sibling, 0 replies; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-20 18:50 UTC (permalink / raw)
To: 'shveta malik' <[email protected]>; +Cc: Peter Smith <[email protected]>; vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Friday, January 20, 2023 12:47 PM shveta malik <[email protected]> wrote:
> 1)
> Tried different variations of altering 'min_apply_delay'. All passed except one
> below:
>
> postgres=# alter subscription mysubnew set (min_apply_delay = '10.9min
> 1ms'); ALTER SUBSCRIPTION postgres=# alter subscription mysubnew set
> (min_apply_delay = '10.9min 2s 1ms'); ALTER SUBSCRIPTION --very similar to
> above but fails, postgres=# alter subscription mysubnew set
> (min_apply_delay = '10.9s 1ms');
> ERROR: invalid input syntax for type interval: "10.9s 1ms"
FYI, this was because the interval type couldn't accept this format.
But now we changed the input format from interval to integer alinged
with recovery_min_apply_delay. Thus, we don't face this issue now.
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-20 18:57 Takamichi Osumi (Fujitsu) <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-20 18:57 UTC (permalink / raw)
To: 'Amit Kapila' <[email protected]>; +Cc: Peter Smith <[email protected]>; vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
Hi,
On Thursday, January 19, 2023 10:17 PM Amit Kapila <[email protected]> wrote:
> On Thu, Jan 19, 2023 at 12:06 PM Takamichi Osumi (Fujitsu)
> <[email protected]> wrote:
> >
> > Kindly have a look at the updated patch v17.
> >
>
> Can we try to optimize the test time for this test? On my machine, it is the
> second highest time-consuming test in src/test/subscription. It seems you are
> waiting twice for apply_delay and both are for streaming cases by varying the
> number of changes. I think it should be just once and that too for the
> non-streaming case. I think it would be good to test streaming code path
> interaction but not sure if it is important enough to have two test cases for
> apply_delay.
The first insert test is for non-streaming case and we need both cases
for coverage. Regarding the time of test, conducted some optimization
such as turning off the initial table sync, shortening the time of wait, and so on.
>
> One minor comment that I observed while going through the patch.
> + /*
> + * The combination of parallel streaming mode and min_apply_delay is
> + not
> + * allowed.
> + */
> + if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
> + opts->min_apply_delay > 0)
>
> I think it would be good if you can specify the reason for not allowing this
> combination in the comments.
Added.
Please have a look at the latest v18 patch in [1].
[1] - https://www.postgresql.org/message-id/TYCPR01MB8373BED9E390C4839AF56685EDC59%40TYCPR01MB8373.jpnprd0...
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-20 19:07 Takamichi Osumi (Fujitsu) <[email protected]>
parent: vignesh C <[email protected]>
1 sibling, 0 replies; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-20 19:07 UTC (permalink / raw)
To: 'vignesh C' <[email protected]>; +Cc: Peter Smith <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
Hi,
On Thursday, January 19, 2023 7:55 PM vignesh C <[email protected]> wrote:
> On Thu, 19 Jan 2023 at 12:06, Takamichi Osumi (Fujitsu)
> <[email protected]> wrote:
> >
> > Updated the comment and the function call.
> >
> > Kindly have a look at the updated patch v17.
>
> Thanks for the updated patch, few comments:
> 1) min_apply_delay was accepting values like '600 m s h', I was not sure if we
> should allow this:
> alter subscription sub1 set (min_apply_delay = ' 600 m s h');
>
> + /*
> + * If no unit was specified, then explicitly
> add 'ms' otherwise
> + * the interval_in function would assume 'seconds'.
> + */
> + if (strspn(tmp, "-0123456789 ") == strlen(tmp))
> + val = psprintf("%sms", tmp);
> + else
> + val = tmp;
> +
> + interval =
> DatumGetIntervalP(DirectFunctionCall3(interval_in,
> +
>
> CStringGetDatum(val),
> +
>
> ObjectIdGetDatum(InvalidOid),
> +
> Int32GetDatum(-1)));
>
FYI, the input can be accepted by the interval type.
Now we changed the direction of the type from interval to integer
but plus some unit can be added like recovery_min_apply_delay.
Please check.
> 3) There is one check at parse_subscription_options and another check in
> AlterSubscription, this looks like a redundant check in case of alter
> subscription, can we try to merge and keep in one place:
> /*
> * The combination of parallel streaming mode and min_apply_delay is not
> * allowed.
> */
> if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
> opts->min_apply_delay > 0)
> {
> if (opts->streaming == LOGICALREP_STREAM_PARALLEL) ereport(ERROR,
> errcode(ERRCODE_SYNTAX_ERROR), errmsg("%s and %s are mutually
> exclusive options",
> "min_apply_delay > 0", "streaming = parallel")); }
>
> if (IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY)) {
> /*
> * The combination of parallel streaming mode and
> * min_apply_delay is not allowed.
> */
> if (opts.min_apply_delay > 0)
> if ((IsSet(opts.specified_opts, SUBOPT_STREAMING) && opts.streaming ==
> LOGICALREP_STREAM_PARALLEL) ||
> (!IsSet(opts.specified_opts, SUBOPT_STREAMING) && sub->stream ==
> LOGICALREP_STREAM_PARALLEL))
> ereport(ERROR,
> errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> errmsg("cannot enable %s for subscription in %s mode",
> "min_apply_delay", "streaming = parallel"));
>
> values[Anum_pg_subscription_subminapplydelay - 1] =
> Int64GetDatum(opts.min_apply_delay);
> replaces[Anum_pg_subscription_subminapplydelay - 1] = true; }
We can't. For create subscription, we need to check the patch
from parse_subscription_options, while for alter subscription,
we need to refer the current MySubscription value for those tests
in AlterSubscription.
> 4) typo "execeeds" should be "exceeds"
>
> + time on the subscriber. Any overhead of time spent in
> logical decoding
> + and in transferring the transaction may reduce the actual wait time.
> + It is also possible that the overhead already execeeds the
> requested
> + <literal>min_apply_delay</literal> value, in which case no
> additional
> + wait is necessary. If the system clocks on publisher and subscriber
> + are not synchronized, this may lead to apply changes earlier
> + than
Fixed.
Kindly have a look at the v18 patch in [1].
[1] - https://www.postgresql.org/message-id/TYCPR01MB8373BED9E390C4839AF56685EDC59%40TYCPR01MB8373.jpnprd0...
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-22 12:42 Takamichi Osumi (Fujitsu) <[email protected]>
parent: Takamichi Osumi (Fujitsu) <[email protected]>
0 siblings, 3 replies; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-22 12:42 UTC (permalink / raw)
To: 'Peter Smith' <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Saturday, January 21, 2023 3:36 AM I wrote:
> Kindly have a look at the patch v18.
I've conducted some refactoring for v18.
Now the latest patch should be tidier and
the comments would be clearer and more aligned as a whole.
Attached the updated patch v19.
Best Regards,
Takamichi Osumi
Attachments:
[application/octet-stream] v19-0001-Time-delayed-logical-replication-subscriber.patch (81.2K, ../../TYCPR01MB8373A23EFC25B22CB132383AEDCB9@TYCPR01MB8373.jpnprd01.prod.outlook.com/2-v19-0001-Time-delayed-logical-replication-subscriber.patch)
download | inline diff:
From 04d5055a353e31f37d8cafca2a672a8f31041193 Mon Sep 17 00:00:00 2001
From: Takamichi Osumi <[email protected]>
Date: Sun, 22 Jan 2023 12:01:26 +0000
Subject: [PATCH v19] Time-delayed logical replication subscriber
Similar to physical replication, a time-delayed copy of the data for
logical replication is useful for some scenarios (particularly to fix
errors that might cause data loss).
This patch implements a new subscription parameter called 'min_apply_delay'.
If the subscription sets min_apply_delay parameter, the logical
replication worker will delay the transaction commit for min_apply_delay
milliseconds.
The delay is calculated between the WAL time stamp and the current time
on the subscriber.
The delay occurs only on WAL records for transaction begins. The main
reason is to avoid keeping a transaction open for a long time. Regular
and prepared transactions are covered. Streamed transactions are also
covered.
The combination of parallel streaming mode and min_apply_delay is not
allowed. The subscriber in the parallel streaming mode applies each
stream on arrival without the time of commit/prepare. So, the
subscriber needs to depend on the arrival time of the stream in this
case, if we apply the time-delayed feature for such transactions. Then
there is a possibility where some unnecessary delay will be added on
the subscriber by network communication break between nodes or other
heavy work load on the publisher. On the other hand, applying the delay
at the end of transaction with parallel apply also can cause issues of
used resource bloat and locks kept in open for a long time. Thus, those
features can't work together.
Author: Euler Taveira
Discussion: https://postgr.es/m/CAB-JLwYOYwL=XTyAXKiH5CtM_Vm8KjKh7aaitCKvmCh4rzr5pQ@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/config.sgml | 13 ++
doc/src/sgml/logical-replication.sgml | 7 +
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 57 +++++-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 7 +-
src/backend/commands/subscriptioncmds.c | 118 +++++++++++-
.../replication/logical/applyparallelworker.c | 3 +-
src/backend/replication/logical/worker.c | 169 ++++++++++++++--
src/bin/pg_dump/pg_dump.c | 15 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 9 +-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_subscription.h | 3 +
src/include/replication/worker_internal.h | 2 +-
src/test/regress/expected/subscription.out | 181 +++++++++++-------
src/test/regress/sql/subscription.sql | 24 +++
src/test/subscription/meson.build | 1 +
src/test/subscription/t/032_apply_delay.pl | 175 +++++++++++++++++
20 files changed, 699 insertions(+), 105 deletions(-)
create mode 100644 src/test/subscription/t/032_apply_delay.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index c1e4048054..bf3c05241c 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7873,6 +7873,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subminapplydelay</structfield> <type>int8</type>
+ </para>
+ <para>
+ The length of time (ms) to delay the application of changes.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subname</structfield> <type>name</type>
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index dc9b78b0b7..b8f6120b0d 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4787,6 +4787,19 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
the <filename>postgresql.conf</filename> file or on the server
command line.
</para>
+ <para>
+ For time-delayed logical replication (i.e. when the subscription is
+ created with parameter min_apply_delay > 0), the apply worker sends a
+ Standby Status Update message to the publisher with a period of
+ <literal>wal_receiver_status_interval</literal>. Make sure to set
+ <literal>wal_receiver_status_interval</literal> less than the
+ <literal>wal_sender_timeout</literal> on the publisher, otherwise, the
+ walsender will repeatedly terminate due to the timeout errors. If
+ <literal>wal_receiver_status_interval</literal> is set to zero, the apply
+ worker doesn't send any feedback messages during the subscriber's
+ <literal>min_apply_delay</literal> period. See
+ <xref linkend="sql-createsubscription"/> for details.
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 1bd5660c87..863af11a47 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -247,6 +247,13 @@
target table.
</para>
+ <para>
+ The subscriber replication can be instructed to lag behind the publisher
+ side changes by specifying the <literal>min_apply_delay</literal>
+ subscription parameter. See <xref linkend="sql-createsubscription"/> for
+ details.
+ </para>
+
<sect2 id="logical-replication-subscription-slot">
<title>Replication Slot Management</title>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index ad93553a1d..1c6e9dd2d1 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -213,8 +213,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
<literal>binary</literal>, <literal>streaming</literal>,
- <literal>disable_on_error</literal>, and
- <literal>origin</literal>.
+ <literal>disable_on_error</literal>,
+ <literal>origin</literal>, and
+ <literal>min_apply_delay</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index eba72c6af6..76ee9c0b3d 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -349,7 +349,45 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
- </variablelist></para>
+
+ <varlistentry>
+ <term><literal>min_apply_delay</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ By default, the subscriber applies changes as soon as possible. This
+ parameter allows the user to delay the application of changes by a
+ given time interval. If the value is specified without units, it is
+ taken as milliseconds. The default is zero (no delay).
+ </para>
+ <para>
+ Any delay occurs only on WAL records for transaction begins after all
+ initial table synchronization has finished. The delay is calculated
+ as the difference between the WAL timestamp as written on the
+ publisher and the current time on the subscriber. Any overhead of
+ time spent in logical decoding and in transferring the transaction
+ may reduce the actual wait time. It is also possible that the overhead
+ already exceeds the requested <literal>min_apply_delay</literal> value,
+ in which case no additional wait is necessary. If the system clocks
+ on publisher and subscriber are not synchronized, this may lead to
+ apply changes earlier than expected, but this is not a major issue
+ because this parameter is typically much larger than the time
+ deviations between servers. Note that if this parameter is set to a
+ long delay, the replication will stop if the replication slot falls
+ behind the current LSN by more than
+ <link linkend="guc-max-slot-wal-keep-size"><literal>max_slot_wal_keep_size</literal></link>.
+ </para>
+ <warning>
+ <para>
+ Delaying the replication can mean there is a much longer time between making
+ a change on the publisher, and that change being committed on the subscriber.
+ This can impact the performance of synchronous replication.
+ See <xref linkend="guc-synchronous-commit"/>.
+ </para>
+ </warning>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
@@ -413,6 +451,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
published with different column lists are not supported.
</para>
+ <para>
+ A non-zero <literal>min_apply_delay</literal> parameter is not allowed when streaming
+ in parallel mode.
+ </para>
+
<para>
We allow non-existent publications to be specified so that users can add
those later. This means
@@ -472,6 +515,18 @@ CREATE SUBSCRIPTION mysub
PUBLICATION insert_only
WITH (enabled = false);
</programlisting></para>
+
+ <para>
+ Create a subscription to a remote server that replicates tables in
+ the <literal>mypub</literal> publication and starts replicating immediately
+ on commit. Pre-existing data is not copied. The application of changes is
+ delayed by 4 hours.
+<programlisting>
+CREATE SUBSCRIPTION mysub
+ CONNECTION 'host=192.0.2.4 port=5432 user=foo dbname=foodb'
+ PUBLICATION mypub
+ WITH (copy_data = false, min_apply_delay = '4h');
+</programlisting></para>
</refsect1>
<refsect1>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index a56ae311c3..c767cc1c3a 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -64,6 +64,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->oid = subid;
sub->dbid = subform->subdbid;
sub->skiplsn = subform->subskiplsn;
+ sub->minapplydelay = subform->subminapplydelay;
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 8608e3fa5b..317c2010cb 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1299,9 +1299,10 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are publicly readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
- subbinary, substream, subtwophasestate, subdisableonerr,
- subslotname, subsynccommit, subpublications, suborigin)
+GRANT SELECT (oid, subdbid, subskiplsn, subminapplydelay, subname, subowner,
+ subenabled, subbinary, substream, subtwophasestate,
+ subdisableonerr, subslotname, subsynccommit, subpublications,
+ suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index baff00dd74..a5d30ec585 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -66,6 +66,7 @@
#define SUBOPT_DISABLE_ON_ERR 0x00000400
#define SUBOPT_LSN 0x00000800
#define SUBOPT_ORIGIN 0x00001000
+#define SUBOPT_MIN_APPLY_DELAY 0x00002000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -90,6 +91,7 @@ typedef struct SubOpts
bool disableonerr;
char *origin;
XLogRecPtr lsn;
+ int min_apply_delay;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -100,7 +102,7 @@ static void check_publications_origin(WalReceiverConn *wrconn,
static void check_duplicates_in_publist(List *publist, Datum *datums);
static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, const char *subname);
static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err);
-
+static int defGetMinApplyDelay(DefElem *def);
/*
* Common option parsing function for CREATE and ALTER SUBSCRIPTION commands.
@@ -146,6 +148,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->disableonerr = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY))
+ opts->min_apply_delay = 0;
/* Parse options */
foreach(lc, stmt_options)
@@ -324,6 +328,12 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ strcmp(defel->defname, "min_apply_delay") == 0)
+ {
+ opts->specified_opts |= SUBOPT_MIN_APPLY_DELAY;
+ opts->min_apply_delay = defGetMinApplyDelay(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -404,6 +414,26 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * The combination of parallel streaming mode and min_apply_delay is not
+ * allowed. The subscriber in the parallel streaming mode applies each
+ * stream on arrival without the time of commit/prepare. So, the
+ * subscriber needs to depend on the arrival time of the stream in this
+ * case, if we apply the time-delayed feature for such transactions. Then
+ * there is a possibility where some unnecessary delay will be added on
+ * the subscriber by network communication break between nodes or other
+ * heavy work load on the publisher. On the other hand, applying the delay
+ * at the end of transaction with parallel apply also can cause issues of
+ * used resource bloat and locks kept in open for a long time. Thus, those
+ * features can't work together.
+ */
+ if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ opts->min_apply_delay > 0 && opts->streaming == LOGICALREP_STREAM_PARALLEL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s and %s are mutually exclusive options",
+ "min_apply_delay > 0", "streaming = parallel"));
}
/*
@@ -560,7 +590,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
- SUBOPT_DISABLE_ON_ERR | SUBOPT_ORIGIN);
+ SUBOPT_DISABLE_ON_ERR | SUBOPT_ORIGIN |
+ SUBOPT_MIN_APPLY_DELAY);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -625,6 +656,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_oid - 1] = ObjectIdGetDatum(subid);
values[Anum_pg_subscription_subdbid - 1] = ObjectIdGetDatum(MyDatabaseId);
values[Anum_pg_subscription_subskiplsn - 1] = LSNGetDatum(InvalidXLogRecPtr);
+ values[Anum_pg_subscription_subminapplydelay - 1] = Int64GetDatum(opts.min_apply_delay);
values[Anum_pg_subscription_subname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
@@ -1054,7 +1086,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
- SUBOPT_ORIGIN);
+ SUBOPT_ORIGIN | SUBOPT_MIN_APPLY_DELAY);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1098,6 +1130,18 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
if (IsSet(opts.specified_opts, SUBOPT_STREAMING))
{
+ /*
+ * The combination of parallel streaming mode and
+ * min_apply_delay is not allowed.
+ */
+ if (opts.streaming == LOGICALREP_STREAM_PARALLEL)
+ if ((IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) && opts.min_apply_delay > 0) ||
+ (!IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) && sub->minapplydelay > 0))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s mode for subscription with %s",
+ "streaming = parallel", "min_apply_delay"));
+
values[Anum_pg_subscription_substream - 1] =
CharGetDatum(opts.streaming);
replaces[Anum_pg_subscription_substream - 1] = true;
@@ -1111,6 +1155,25 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
= true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY))
+ {
+ /*
+ * The combination of parallel streaming mode and
+ * min_apply_delay is not allowed.
+ */
+ if (opts.min_apply_delay > 0)
+ if ((IsSet(opts.specified_opts, SUBOPT_STREAMING) && opts.streaming == LOGICALREP_STREAM_PARALLEL) ||
+ (!IsSet(opts.specified_opts, SUBOPT_STREAMING) && sub->stream == LOGICALREP_STREAM_PARALLEL))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s for subscription in %s mode",
+ "min_apply_delay", "streaming = parallel"));
+
+ values[Anum_pg_subscription_subminapplydelay - 1] =
+ Int64GetDatum(opts.min_apply_delay);
+ replaces[Anum_pg_subscription_subminapplydelay - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
@@ -2185,3 +2248,52 @@ defGetStreamingMode(DefElem *def)
def->defname)));
return LOGICALREP_STREAM_OFF; /* keep compiler quiet */
}
+
+
+/*
+ * Extract the min_apply_delay mode value from a DefElem. This is very similar
+ * to PGC_INT case of parse_and_validate_value(), because min_apply_delay
+ * accepts the same string as recovery_min_apply_delay.
+ */
+int
+defGetMinApplyDelay(DefElem *def)
+{
+ char *value;
+ int result;
+ const char *hintmsg;
+
+ /*
+ * Raise an ERROR if no parameter value given
+ */
+ if (def->arg == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s requires an integer value",
+ def->defname)));
+
+ value = defGetString(def);
+
+ /*
+ * Parse given string as parameter which has millisecond unit
+ */
+ if (!parse_int(value, &result, GUC_UNIT_MS, &hintmsg))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid value for parameter \"%s\": \"%s\"",
+ "min_apply_delay", value),
+ hintmsg ? errhint("%s", _(hintmsg)) : 0));
+
+ /*
+ * Check lower bound. parse_int() has been already confirmed that result
+ * is equal to or smaller than INT_MAX.
+ */
+ if (result < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d ms is outside the valid range for parameter \"%s\" (%d .. %d)",
+ result,
+ "min_apply_delay",
+ 0, INT_MAX)));
+
+ return result;
+}
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index 3579e704fe..7302bce7a0 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -704,7 +704,8 @@ pa_process_spooled_messages_if_required(void)
{
apply_spooled_messages(&MyParallelShared->fileset,
MyParallelShared->xid,
- InvalidXLogRecPtr);
+ InvalidXLogRecPtr,
+ 0);
pa_set_fileset_state(MyParallelShared, FS_EMPTY);
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a0084c7ef6..31719db030 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -318,6 +318,17 @@ static List *on_commit_wakeup_workers_subids = NIL;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/*
+ * In order to avoid walsender timeout for time-delayed replication the worker
+ * process keeps sending feedback messages during the delay period.
+ * Meanwhile, the feature delays the apply before starting the
+ * transaction and thus we don't write WALs for the suspended changes during
+ * the wait. When the worker process sends a feedback message
+ * during the delay, we should not make positions of the flushed and apply LSN
+ * overwritten by the last received latest LSN. See send_feedback() for details.
+ */
+static XLogRecPtr last_received = InvalidXLogRecPtr;
+
/* fields valid only when processing streamed transaction */
static bool in_streamed_transaction = false;
@@ -388,10 +399,13 @@ static void stream_write_change(char action, StringInfo s);
static void stream_open_and_write_change(TransactionId xid, char action, StringInfo s);
static void stream_close_file(void);
-static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
+static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply,
+ bool in_delayed_apply);
static void DisableSubscriptionAndExit(void);
+static void maybe_delay_apply(TransactionId xid, TimestampTz finish_ts);
+
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
@@ -998,6 +1012,108 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
ExecStoreVirtualTuple(slot);
}
+/*
+ * When min_apply_delay parameter is set on the subscriber, we wait long enough
+ * to make sure a transaction is applied at least that interval behind the
+ * publisher.
+ *
+ * While the physical replication applies the delay at commit time, this
+ * feature applies the delay for the next transaction but before starting the
+ * transaction. This is mainly because keeping a transaction that conducted
+ * write operations open for a long time results in some issues such as bloat
+ * and locks.
+ *
+ * The min_apply_delay parameter will take effect only after all tables are in
+ * READY state.
+ *
+ * xid is the transaction id where we apply the delay.
+ *
+ * finish_ts is the commit/prepare time of both regular (non-streamed) and
+ * streamed transactions. Unlike the regular (non-streamed) cases, the delay
+ * is applied in a STREAM COMMIT/STREAM PREPARE message for streamed
+ * transactions. The STREAM START message does not contain a commit/prepare
+ * time (it will be available when the in-progress transaction finishes).
+ * Hence, it's not appropriate to apply a delay at the STREAM START time.
+ */
+static void
+maybe_delay_apply(TransactionId xid, TimestampTz finish_ts)
+{
+ Assert(finish_ts > 0);
+
+ /* Nothing to do if no delay set */
+ if (!MySubscription->minapplydelay)
+ return;
+
+ /*
+ * The min_apply_delay parameter is ignored until all tablesync workers
+ * have reached READY state. This is because if we allowed the delay
+ * during the catchup phase, then once we reached the limit of tablesync
+ * workers it would impose a delay for each subsequent worker. That would
+ * cause initial table synchronization completion to take a long time.
+ */
+ if (!AllTablesyncsReady())
+ return;
+
+ /* Apply the delay by the latch mechanism */
+ while (true)
+ {
+ long diffms;
+
+ ResetLatch(MyLatch);
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* This might change wal_receiver_status_interval */
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ /*
+ * Before calculating the time duration, reload the catalog if needed.
+ */
+ if (!in_remote_transaction && !in_streamed_transaction)
+ {
+ AcceptInvalidationMessages();
+ maybe_reread_subscription();
+ }
+
+ diffms = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ TimestampTzPlusMilliseconds(finish_ts, MySubscription->minapplydelay));
+
+ /*
+ * Exit without arming the latch if it's already past time to apply
+ * this transaction.
+ */
+ if (diffms <= 0)
+ break;
+
+ elog(DEBUG2, "time-delayed replication for txid %u, min_apply_delay = %lld ms, Remaining wait time: %ld ms",
+ xid, (long long) MySubscription->minapplydelay, diffms);
+
+ /*
+ * Call send_feedback() to prevent the publisher from exiting by
+ * timeout during the delay, when wal_receiver_status_interval is
+ * available.
+ */
+ if (wal_receiver_status_interval > 0
+ && diffms > wal_receiver_status_interval * 1000)
+ {
+ WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ (long) wal_receiver_status_interval * 1000,
+ WAIT_EVENT_RECOVERY_APPLY_DELAY);
+ send_feedback(last_received, true, false, true);
+ }
+ else
+ WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ diffms,
+ WAIT_EVENT_RECOVERY_APPLY_DELAY);
+ }
+}
+
/*
* Handle BEGIN message.
*/
@@ -1012,6 +1128,9 @@ apply_handle_begin(StringInfo s)
logicalrep_read_begin(s, &begin_data);
set_apply_error_context_xact(begin_data.xid, begin_data.final_lsn);
+ /* Should we delay the current transaction? */
+ maybe_delay_apply(begin_data.xid, begin_data.committime);
+
remote_final_lsn = begin_data.final_lsn;
maybe_start_skipping_changes(begin_data.final_lsn);
@@ -1069,6 +1188,9 @@ apply_handle_begin_prepare(StringInfo s)
logicalrep_read_begin_prepare(s, &begin_data);
set_apply_error_context_xact(begin_data.xid, begin_data.prepare_lsn);
+ /* Should we delay the current prepared transaction? */
+ maybe_delay_apply(begin_data.xid, begin_data.prepare_time);
+
remote_final_lsn = begin_data.prepare_lsn;
maybe_start_skipping_changes(begin_data.prepare_lsn);
@@ -1316,7 +1438,8 @@ apply_handle_stream_prepare(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset,
- prepare_data.xid, prepare_data.prepare_lsn);
+ prepare_data.xid, prepare_data.prepare_lsn,
+ prepare_data.prepare_time);
/* Mark the transaction as prepared. */
apply_handle_prepare_internal(&prepare_data);
@@ -2010,10 +2133,13 @@ ensure_last_message(FileSet *stream_fileset, TransactionId xid, int fileno,
/*
* Common spoolfile processing.
+ *
+ * The commit/prepare time for streaming transaction is required to achieve
+ * time-delayed replication.
*/
void
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn, TimestampTz finish_ts)
{
StringInfoData s2;
int nchanges;
@@ -2024,6 +2150,10 @@ apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
int fileno;
off_t offset;
+ /* Should we delay the current transaction? */
+ if (finish_ts)
+ maybe_delay_apply(xid, finish_ts);
+
if (!am_parallel_apply_worker())
maybe_start_skipping_changes(lsn);
@@ -2173,7 +2303,7 @@ apply_handle_stream_commit(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset, xid,
- commit_data.commit_lsn);
+ commit_data.commit_lsn, commit_data.committime);
apply_handle_commit_internal(&commit_data);
@@ -3446,7 +3576,7 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
* Apply main loop.
*/
static void
-LogicalRepApplyLoop(XLogRecPtr last_received)
+LogicalRepApplyLoop(void)
{
TimestampTz last_recv_timestamp = GetCurrentTimestamp();
bool ping_sent = false;
@@ -3567,7 +3697,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
if (last_received < end_lsn)
last_received = end_lsn;
- send_feedback(last_received, reply_requested, false);
+ send_feedback(last_received, reply_requested, false, false);
UpdateWorkerStats(last_received, timestamp, true);
}
/* other message types are purposefully ignored */
@@ -3580,7 +3710,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
/* confirm all writes so far */
- send_feedback(last_received, false, false);
+ send_feedback(last_received, false, false, false);
if (!in_remote_transaction && !in_streamed_transaction)
{
@@ -3677,7 +3807,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
}
- send_feedback(last_received, requestReply, requestReply);
+ send_feedback(last_received, requestReply, requestReply, false);
/*
* Force reporting to ensure long idle periods don't lead to
@@ -3707,7 +3837,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
* to send a response to avoid timeouts.
*/
static void
-send_feedback(XLogRecPtr recvpos, bool force, bool requestReply)
+send_feedback(XLogRecPtr recvpos, bool force, bool requestReply, bool in_delayed_apply)
{
static StringInfo reply_message = NULL;
static TimestampTz send_time = 0;
@@ -3737,8 +3867,15 @@ send_feedback(XLogRecPtr recvpos, bool force, bool requestReply)
/*
* No outstanding transactions to flush, we can report the latest received
* position. This is important for synchronous replication.
+ *
+ * If the subscriber side apply is delayed (because of time-delayed
+ * replication) then do not tell the publisher that the received latest
+ * LSN is already applied and flushed, otherwise, it leads to the
+ * publisher side making a wrong assumption of logical replication
+ * progress. Instead, we just send a feedback message to avoid a publisher
+ * timeout during the delay.
*/
- if (!have_pending_txes)
+ if (!have_pending_txes && !in_delayed_apply)
flushpos = writepos = recvpos;
if (writepos < last_writepos)
@@ -3775,11 +3912,12 @@ send_feedback(XLogRecPtr recvpos, bool force, bool requestReply)
pq_sendint64(reply_message, now); /* sendTime */
pq_sendbyte(reply_message, requestReply); /* replyRequested */
- elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write %X/%X, flush %X/%X",
+ elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write %X/%X, flush %X/%X in-delayed: %d",
force,
LSN_FORMAT_ARGS(recvpos),
LSN_FORMAT_ARGS(writepos),
- LSN_FORMAT_ARGS(flushpos));
+ LSN_FORMAT_ARGS(flushpos),
+ in_delayed_apply);
walrcv_send(LogRepWorkerWalRcvConn,
reply_message->data, reply_message->len);
@@ -4354,11 +4492,11 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname)
* of system resource error and are not repeatable.
*/
static void
-start_apply(XLogRecPtr origin_startpos)
+start_apply(void)
{
PG_TRY();
{
- LogicalRepApplyLoop(origin_startpos);
+ LogicalRepApplyLoop();
}
PG_CATCH();
{
@@ -4645,7 +4783,8 @@ ApplyWorkerMain(Datum main_arg)
}
/* Run the main loop. */
- start_apply(origin_startpos);
+ last_received = origin_startpos;
+ start_apply();
proc_exit(0);
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 527c7651ab..c0f69cb43b 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4494,6 +4494,7 @@ getSubscriptions(Archive *fout)
int i_subsynccommit;
int i_subpublications;
int i_subbinary;
+ int i_subminapplydelay;
int i,
ntups;
@@ -4546,9 +4547,13 @@ getSubscriptions(Archive *fout)
LOGICALREP_TWOPHASE_STATE_DISABLED);
if (fout->remoteVersion >= 160000)
- appendPQExpBufferStr(query, " s.suborigin\n");
+ appendPQExpBufferStr(query,
+ " s.suborigin,\n"
+ " s.subminapplydelay\n");
else
- appendPQExpBuffer(query, " '%s' AS suborigin\n", LOGICALREP_ORIGIN_ANY);
+ appendPQExpBuffer(query, " '%s' AS suborigin,\n"
+ " 0 AS subminapplydelay\n",
+ LOGICALREP_ORIGIN_ANY);
appendPQExpBufferStr(query,
"FROM pg_subscription s\n"
@@ -4576,6 +4581,7 @@ getSubscriptions(Archive *fout)
i_subtwophasestate = PQfnumber(res, "subtwophasestate");
i_subdisableonerr = PQfnumber(res, "subdisableonerr");
i_suborigin = PQfnumber(res, "suborigin");
+ i_subminapplydelay = PQfnumber(res, "subminapplydelay");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4606,6 +4612,8 @@ getSubscriptions(Archive *fout)
subinfo[i].subdisableonerr =
pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
subinfo[i].suborigin = pg_strdup(PQgetvalue(res, i, i_suborigin));
+ subinfo[i].subminapplydelay =
+ strtoi64(PQgetvalue(res, i, i_subminapplydelay), NULL, 10);
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -4687,6 +4695,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
+ if (subinfo->subminapplydelay > 0)
+ appendPQExpBuffer(query, ", min_apply_delay = '" INT64_FORMAT " ms'", subinfo->subminapplydelay);
+
appendPQExpBufferStr(query, ");\n");
if (subinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index e7cbd8d7ed..e2525f70ab 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -661,6 +661,7 @@ typedef struct _SubscriptionInfo
char *subdisableonerr;
char *suborigin;
char *subsynccommit;
+ int64 subminapplydelay;
char *subpublications;
} SubscriptionInfo;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index c8a0bb7b3a..8a27063bed 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6472,7 +6472,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false, false, false, false, false, false, false};
+ false, false, false, false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6527,10 +6527,13 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Two-phase commit"),
gettext_noop("Disable on error"));
+ /* Origin and min_apply_delay are only supported in v16 and higher */
if (pset.sversion >= 160000)
appendPQExpBuffer(&buf,
- ", suborigin AS \"%s\"\n",
- gettext_noop("Origin"));
+ ", suborigin AS \"%s\"\n"
+ ", subminapplydelay AS \"%s\"\n",
+ gettext_noop("Origin"),
+ gettext_noop("Min apply delay (ms)"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 5e1882eaea..e8b9a43a47 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1925,7 +1925,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "origin", "slot_name",
+ COMPLETE_WITH("binary", "disable_on_error", "min_apply_delay", "origin", "slot_name",
"streaming", "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
@@ -3268,7 +3268,7 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "origin", "slot_name",
+ "disable_on_error", "enabled", "min_apply_delay", "origin", "slot_name",
"streaming", "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index b0f2a1705d..e06f35c037 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -70,6 +70,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
XLogRecPtr subskiplsn; /* All changes finished at this LSN are
* skipped */
+ int64 subminapplydelay; /* Replication apply delay (ms) */
+
NameData subname; /* Name of the subscription */
Oid subowner BKI_LOOKUP(pg_authid); /* Owner of the subscription */
@@ -120,6 +122,7 @@ typedef struct Subscription
* in */
XLogRecPtr skiplsn; /* All changes finished at this LSN are
* skipped */
+ int64 minapplydelay; /* Replication apply delay (ms) */
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index dc87a4edd1..3dc09d1a4c 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -255,7 +255,7 @@ extern void stream_stop_internal(TransactionId xid);
/* Common streaming function to apply all the spooled messages */
extern void apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn);
+ XLogRecPtr lsn, TimestampTz finish_ts);
extern void apply_dispatch(StringInfo s);
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 4e5cb0d3a9..1230bcb096 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -114,18 +114,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -135,10 +135,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -155,10 +155,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -167,10 +167,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -202,10 +202,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -239,19 +239,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -263,27 +263,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -298,10 +298,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -316,10 +316,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -355,10 +355,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -367,10 +367,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -380,10 +380,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,20 +396,57 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail -- min_apply_delay must be a non-negative integer
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = foo);
+ERROR: invalid value for parameter "min_apply_delay": "foo"
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = -1);
+ERROR: -1 ms is outside the valid range for parameter "min_apply_delay" (0 .. 2147483647)
+-- fail - utilizing streaming = parallel with time-delayed replication is not supported
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = parallel, min_apply_delay = 123);
+ERROR: min_apply_delay > 0 and streaming = parallel are mutually exclusive options
+-- success -- min_apply_delay value without unit is taken as milliseconds
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = 123);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 123 | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+-- success -- min_apply_delay value with unit is converted into ms and stored as an integer
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = '1 d');
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 86400000 | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+-- fail - alter subscription with streaming = parallel should fail when time-delayed replication is set
+ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
+ERROR: cannot enable streaming = parallel mode for subscription with min_apply_delay
+-- fail - alter subscription with min_apply_delay should fail when streaming = parallel is set
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 0, streaming = parallel);
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 123);
+ERROR: cannot enable min_apply_delay for subscription in streaming = parallel mode
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
RESET SESSION AUTHORIZATION;
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 5f27b7d776..53fe2a4c6b 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -279,6 +279,30 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail -- min_apply_delay must be a non-negative integer
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = foo);
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = -1);
+
+-- fail - utilizing streaming = parallel with time-delayed replication is not supported
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = parallel, min_apply_delay = 123);
+
+-- success -- min_apply_delay value without unit is taken as milliseconds
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, min_apply_delay = 123);
+\dRs+
+
+-- success -- min_apply_delay value with unit is converted into ms and stored as an integer
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = '1 d');
+\dRs+
+
+-- fail - alter subscription with streaming = parallel should fail when time-delayed replication is set
+ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
+
+-- fail - alter subscription with min_apply_delay should fail when streaming = parallel is set
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 0, streaming = parallel);
+ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = 123);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index 3db0fdfd96..a186876eb4 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -38,6 +38,7 @@ tests += {
't/029_on_error.pl',
't/030_origin.pl',
't/031_column_list.pl',
+ 't/032_apply_delay.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/032_apply_delay.pl b/src/test/subscription/t/032_apply_delay.pl
new file mode 100644
index 0000000000..37388e474f
--- /dev/null
+++ b/src/test/subscription/t/032_apply_delay.pl
@@ -0,0 +1,175 @@
+
+# Copyright (c) 2023, PostgreSQL Global Development Group
+
+# Test replication apply delay
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Confirm the time-delayed replication has been effective from the server log
+# message where the apply worker emits for applying delay. Moreover, verifies
+# that the current worker's delayed time is sufficiently bigger than the
+# expected value, in order to check any update of the min_apply_delay.
+sub check_apply_delay_log
+{
+ my ($node_subscriber, $offset, $expected) = @_;
+
+ my $log_location = $node_subscriber->wait_for_log(qr/time-delayed replication for txid (\d+), min_apply_delay = (\d+) ms, Remaining wait time: (\d+) ms/, $offset);
+
+ cmp_ok($log_location, '>', $offset,
+ "logfile contains triggered logical replication apply delay"
+ );
+
+ # Get the delay time from the server log
+ my $contents = slurp_file($node_subscriber->logfile, $offset);
+ $contents =~
+ qr/time-delayed replication for txid (\d+), min_apply_delay = (\d+) ms, Remaining wait time: (\d+) ms/
+ or die "could not get the apply worker wait time";
+ my $logged_delay = $3;
+
+ # Is it larger than expected?
+ cmp_ok($logged_delay, '>', $expected,
+ "The apply worker wait time has expected duration"
+ );
+}
+
+# Compare inserted time on the publisher with applied time on the subscriber to
+# confirm the latter is applied after expected time. The time is automatically
+# generated and stored in the table column 'c'.
+sub check_apply_delay_time
+{
+ my ($node_publisher, $node_subscriber, $primary_key, $expected_diffs) = @_;
+
+ my $inserted_time_on_pub = $node_publisher->safe_psql('postgres', qq[
+ SELECT extract(epoch from c) FROM test_tab WHERE a = $primary_key;
+ ]);
+
+ my $inserted_time_on_sub = $node_subscriber->safe_psql('postgres', qq[
+ SELECT extract(epoch from c) FROM test_tab WHERE a = $primary_key;
+ ]);
+
+ cmp_ok($inserted_time_on_sub - $inserted_time_on_pub, '>', $expected_diffs,
+ "The tuple on the subscriber was modified later than the publisher");
+}
+
+# Create publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = debug2");
+$node_subscriber->start;
+
+# Setup structure on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar, c timestamptz (6) DEFAULT now())");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b text, c timestamptz (6) DEFAULT now())"
+);
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+# The column 'c' must not be published because we want to compare the time
+# difference.
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE test_tab (a, b)");
+
+my $appname = 'tap_sub';
+
+# Create a subscription that applies the trasaction after 50 milliseconds delay
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (copy_data = off, min_apply_delay = '50ms', streaming = 'on')"
+);
+
+# New row to trigger apply delay
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (2, 'bar')");
+
+$node_publisher->wait_for_catchup($appname);
+
+my $result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM test_tab");
+is($result, qq(2|1|2), 'check if the new rows were applied to subscriber');
+
+# Note that we cannot call check_apply_delay_log() here because there is a
+# possibility that the delay is skipped. The event happens when the WAL
+# replication between publisher and subscriber is delayed due to a mechanical
+# problem. The log output will be checked later - substantial delay-time case.
+
+# Verify that the subscriber lags the publisher by at least 50 milliseconds
+check_apply_delay_time($node_publisher, $node_subscriber, '2', '0.05');
+
+# Setup for streaming case
+$node_publisher->append_conf('postgres.conf',
+ 'logical_decoding_mode = immediate');
+$node_publisher->reload;
+
+# Run a query to make sure that the reload has taken effect.
+$node_publisher->safe_psql('postgres', q{SELECT 1});
+
+# Test streamed transaction by insert
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5) s(i);");
+
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM test_tab");
+is($result, qq(5|1|5), 'check if the new rows were applied to subscriber');
+
+# Verify that the subscriber lags the publisher by at least 50 milliseconds
+check_apply_delay_time($node_publisher, $node_subscriber, '5', '0.05');
+
+# Test whether ALTER SUBSCRIPTION changes the delayed time of the apply worker
+# (1 day 5 minutes). Note that the extra 5 minute is to account for any
+# decoding/network overhead.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (min_apply_delay = 86700000)"
+);
+
+# Check log starting now for logical replication apply delay
+my $offset = -s $node_subscriber->logfile;
+
+# New row to trigger apply delay
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (0, 'foobar')");
+
+# Make sure the apply worker knows to wait for more than 1 day
+check_apply_delay_log($node_subscriber, $offset, "86400000");
+
+# Disable subscription and the worker should die immediately
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub DISABLE;"
+);
+
+# Wait until worker dies
+my $sub_query =
+ "SELECT count(1) = 0 FROM pg_stat_subscription WHERE subname = 'tap_sub' AND pid IS NOT NULL;";
+$node_subscriber->poll_query_until('postgres', $sub_query)
+ or die "Timed out while waiting for subscriber to die";
+
+# Confirm disabling the subscription by ALTER SUBSCRIPTION DISABLE did not cause
+# the delayed transaction to be applied.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(a) FROM test_tab WHERE a = 0;");
+is($result, qq(0), "check the delayed transaction was not applied");
+
+$node_subscriber->stop;
+$node_publisher->stop;
+
+done_testing();
--
2.30.0
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-23 08:06 Peter Smith <[email protected]>
parent: Takamichi Osumi (Fujitsu) <[email protected]>
2 siblings, 2 replies; 40+ messages in thread
From: Peter Smith @ 2023-01-23 08:06 UTC (permalink / raw)
To: Takamichi Osumi (Fujitsu) <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
Here are my review comments for v19-0001.
======
Commit message
1.
The combination of parallel streaming mode and min_apply_delay is not
allowed. The subscriber in the parallel streaming mode applies each
stream on arrival without the time of commit/prepare. So, the
subscriber needs to depend on the arrival time of the stream in this
case, if we apply the time-delayed feature for such transactions. Then
there is a possibility where some unnecessary delay will be added on
the subscriber by network communication break between nodes or other
heavy work load on the publisher. On the other hand, applying the delay
at the end of transaction with parallel apply also can cause issues of
used resource bloat and locks kept in open for a long time. Thus, those
features can't work together.
~
I think the above is just cut/paste from a code comment within
subscriptioncmds.c. See review comments #5 below -- so if the code is
changed then this commit message should also change to match it.
======
doc/src/sgml/ref/create_subscription.sgml
2.
+ <varlistentry>
+ <term><literal>min_apply_delay</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ By default, the subscriber applies changes as soon as possible. This
+ parameter allows the user to delay the application of changes by a
+ given time interval. If the value is specified without units, it is
+ taken as milliseconds. The default is zero (no delay).
+ </para>
2a.
The pgdocs says this is an integer default to “ms” unit. Also, the
example on this same page shows it is set to '4h'. But I did not see
any mention of what other units are available to the user. Maybe other
time units should be mentioned here, or maybe a link should be given
to the section “20.1.1. Parameter Names and Values".
~
2b.
Previously the word "interval" was deliberately used because this
parameter had interval support. But maybe now it should be changed so
it is not misleading.
"a given time interval" --> "a given time period" ??
======
src/backend/commands/subscriptioncmds.c
3. Forward declare
+static int defGetMinApplyDelay(DefElem *def);
If the new function is implemented as static near the top of this
source file then this forward declare would not even be necessary,
right?
~~~
4. parse_subscription_options
@@ -324,6 +328,12 @@ parse_subscription_options(ParseState *pstate,
List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ strcmp(defel->defname, "min_apply_delay") == 0)
+ {
+ opts->specified_opts |= SUBOPT_MIN_APPLY_DELAY;
+ opts->min_apply_delay = defGetMinApplyDelay(defel);
+ }
Should this code fragment be calling errorConflictingDefElem so it
will report an error if the same min_apply_delay parameter is
redundantly repeated? (IIUC, this appears to be the code pattern for
other parameters nearby).
~~~
5. parse_subscription_options
+ /*
+ * The combination of parallel streaming mode and min_apply_delay is not
+ * allowed. The subscriber in the parallel streaming mode applies each
+ * stream on arrival without the time of commit/prepare. So, the
+ * subscriber needs to depend on the arrival time of the stream in this
+ * case, if we apply the time-delayed feature for such transactions. Then
+ * there is a possibility where some unnecessary delay will be added on
+ * the subscriber by network communication break between nodes or other
+ * heavy work load on the publisher. On the other hand, applying the delay
+ * at the end of transaction with parallel apply also can cause issues of
+ * used resource bloat and locks kept in open for a long time. Thus, those
+ * features can't work together.
+ */
IMO some re-wording might be warranted here. I am not sure quite how
to do it. Perhaps like below?
SUGGESTION
The combination of parallel streaming mode and min_apply_delay is not allowed.
Here are some reasons why these features are incompatible:
a. In the parallel streaming mode the subscriber applies each stream
on arrival without knowledge of the commit/prepare time. This means we
cannot calculate the underlying network/decoding lag between publisher
and subscriber, and so always waiting for the full 'min_apply_delay'
period might include unnecessary delay.
b. If we apply the delay at the end of the transaction of the parallel
apply then that would cause issues related to resource bloat and locks
being held for a long time.
~~~
6. defGetMinApplyDelay
+
+
+/*
+ * Extract the min_apply_delay mode value from a DefElem. This is very similar
+ * to PGC_INT case of parse_and_validate_value(), because min_apply_delay
+ * accepts the same string as recovery_min_apply_delay.
+ */
+int
+defGetMinApplyDelay(DefElem *def)
6a.
"same string" -> "same parameter format" ??
~
6b.
I thought this function should be implemented as static and located at
the top of the subscriptioncmds.c source file.
======
src/backend/replication/logical/worker.c
7. maybe_delay_apply
+static void maybe_delay_apply(TransactionId xid, TimestampTz finish_ts);
Is there a reason why this is here? AFAIK the static implementation
precedes any usage so I doubt this forward declaration is required.
~~~
8. send_feedback
@@ -3775,11 +3912,12 @@ send_feedback(XLogRecPtr recvpos, bool force,
bool requestReply)
pq_sendint64(reply_message, now); /* sendTime */
pq_sendbyte(reply_message, requestReply); /* replyRequested */
- elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write
%X/%X, flush %X/%X",
+ elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write
%X/%X, flush %X/%X in-delayed: %d",
force,
LSN_FORMAT_ARGS(recvpos),
LSN_FORMAT_ARGS(writepos),
- LSN_FORMAT_ARGS(flushpos));
+ LSN_FORMAT_ARGS(flushpos),
+ in_delayed_apply);
Wondering if it is better to write this as:
"sending feedback (force %d, in_delayed_apply %d) to recv %X/%X, write
%X/%X, flush %X/%X"
======
src/test/regress/sql/subscription.sql
9. Add new test?
Should there be an additional test to check redundant parameter
setting -- eg. "... WITH (min_apply_delay=123, min_apply_delay=456)"
(this is related to the review comment #4)
~
10. Add new tests?
Should there be other tests just to verify different units (like 'd',
'h', 'min') are working OK?
======
src/test/subscription/t/032_apply_delay.pl
11.
+# Confirm the time-delayed replication has been effective from the server log
+# message where the apply worker emits for applying delay. Moreover, verifies
+# that the current worker's delayed time is sufficiently bigger than the
+# expected value, in order to check any update of the min_apply_delay.
+sub check_apply_delay_log
"the current worker's delayed time..." --> "the current worker's
remaining wait time..." ??
~~~
12.
+ # Get the delay time from the server log
+ my $contents = slurp_file($node_subscriber->logfile, $offset);
"Get the delay time...." --> "Get the remaining wait time..."
~~~
13.
+# Create a subscription that applies the trasaction after 50 milliseconds delay
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr
application_name=$appname' PUBLICATION tap_pub WITH (copy_data = off,
min_apply_delay = '50ms', streaming = 'on')"
+);
13a.
typo: "trasaction"
~
13b
50ms seems an extremely short time – How do you even know if this is
testing anything related to the time delay? You may just be detecting
the normal lag between publisher and subscriber without time delay
having much to do with anything.
~
14.
+# Note that we cannot call check_apply_delay_log() here because there is a
+# possibility that the delay is skipped. The event happens when the WAL
+# replication between publisher and subscriber is delayed due to a mechanical
+# problem. The log output will be checked later - substantial delay-time case.
+
+# Verify that the subscriber lags the publisher by at least 50 milliseconds
+check_apply_delay_time($node_publisher, $node_subscriber, '2', '0.05');
14a.
"The event happens..." ??
Did you mean "This might happen if the WAL..."
~
14b.
The log output will be checked later - substantial delay-time case.
I think that needs re-wording to clarify.
e.g1. you have nothing called a "substantial delay-time" case.
e.g2. the word "later" confused me. Originally, I thought you meant it
is not tested yet but that you will check it "later", but now IIUC you
are just referring to the "1 day 5 minutes" test that comes below in
this location TAP file (??)
------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-23 10:44 Amit Kapila <[email protected]>
parent: Peter Smith <[email protected]>
1 sibling, 2 replies; 40+ messages in thread
From: Amit Kapila @ 2023-01-23 10:44 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Takamichi Osumi (Fujitsu) <[email protected]>; vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Mon, Jan 23, 2023 at 1:36 PM Peter Smith <[email protected]> wrote:
>
> Here are my review comments for v19-0001.
>
...
>
> 5. parse_subscription_options
>
> + /*
> + * The combination of parallel streaming mode and min_apply_delay is not
> + * allowed. The subscriber in the parallel streaming mode applies each
> + * stream on arrival without the time of commit/prepare. So, the
> + * subscriber needs to depend on the arrival time of the stream in this
> + * case, if we apply the time-delayed feature for such transactions. Then
> + * there is a possibility where some unnecessary delay will be added on
> + * the subscriber by network communication break between nodes or other
> + * heavy work load on the publisher. On the other hand, applying the delay
> + * at the end of transaction with parallel apply also can cause issues of
> + * used resource bloat and locks kept in open for a long time. Thus, those
> + * features can't work together.
> + */
>
> IMO some re-wording might be warranted here. I am not sure quite how
> to do it. Perhaps like below?
>
> SUGGESTION
>
> The combination of parallel streaming mode and min_apply_delay is not allowed.
>
> Here are some reasons why these features are incompatible:
> a. In the parallel streaming mode the subscriber applies each stream
> on arrival without knowledge of the commit/prepare time. This means we
> cannot calculate the underlying network/decoding lag between publisher
> and subscriber, and so always waiting for the full 'min_apply_delay'
> period might include unnecessary delay.
> b. If we apply the delay at the end of the transaction of the parallel
> apply then that would cause issues related to resource bloat and locks
> being held for a long time.
>
> ~~~
>
How about something like:
The combination of parallel streaming mode and min_apply_delay is not
allowed. This is because we start applying the transaction stream as
soon as the first change arrives without knowing the transaction's
prepare/commit time. This means we cannot calculate the underlying
network/decoding lag between publisher and subscriber, and so always
waiting for the full 'min_apply_delay' period might include
unnecessary delay.
The other possibility is to apply the delay at the end of the parallel
apply transaction but that would cause issues related to resource
bloat and locks being held for a long time.
> 6. defGetMinApplyDelay
>
> +
> +
> +/*
> + * Extract the min_apply_delay mode value from a DefElem. This is very similar
> + * to PGC_INT case of parse_and_validate_value(), because min_apply_delay
> + * accepts the same string as recovery_min_apply_delay.
> + */
> +int
> +defGetMinApplyDelay(DefElem *def)
>
> 6a.
> "same string" -> "same parameter format" ??
>
> ~
>
> 6b.
> I thought this function should be implemented as static and located at
> the top of the subscriptioncmds.c source file.
>
I agree that this should be a static function but I think its current
location is a better place as other similar function is just above it.
>
> ======
> src/test/regress/sql/subscription.sql
>
> 9. Add new test?
>
> Should there be an additional test to check redundant parameter
> setting -- eg. "... WITH (min_apply_delay=123, min_apply_delay=456)"
>
I don't think that will be of much help. We don't seem to have other
tests for subscription parameters.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-23 12:06 Amit Kapila <[email protected]>
parent: Takamichi Osumi (Fujitsu) <[email protected]>
2 siblings, 1 reply; 40+ messages in thread
From: Amit Kapila @ 2023-01-23 12:06 UTC (permalink / raw)
To: Takamichi Osumi (Fujitsu) <[email protected]>; +Cc: Peter Smith <[email protected]>; vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Sun, Jan 22, 2023 at 6:12 PM Takamichi Osumi (Fujitsu)
<[email protected]> wrote:
>
>
> Attached the updated patch v19.
>
Few comments:
=============
1.
}
+
+
+/*
Only one empty line is sufficient between different functions.
2.
+ if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
+ opts->min_apply_delay > 0 && opts->streaming == LOGICALREP_STREAM_PARALLEL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s and %s are mutually exclusive options",
+ "min_apply_delay > 0", "streaming = parallel"));
}
I think here we should add a comment for the translator as we are
doing in some other nearby cases.
3.
+ /*
+ * The combination of parallel streaming mode and
+ * min_apply_delay is not allowed.
+ */
+ if (opts.streaming == LOGICALREP_STREAM_PARALLEL)
+ if ((IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) &&
opts.min_apply_delay > 0) ||
+ (!IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) &&
sub->minapplydelay > 0))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s mode for subscription with %s",
+ "streaming = parallel", "min_apply_delay"));
+
A. When can second condition ((!IsSet(opts.specified_opts,
SUBOPT_MIN_APPLY_DELAY) && sub->minapplydelay > 0)) in above check be
true?
B. In comments, you can say "See parse_subscription_options."
4.
+/*
+ * When min_apply_delay parameter is set on the subscriber, we wait long enough
+ * to make sure a transaction is applied at least that interval behind the
+ * publisher.
Shouldn't this part of the comment needs to be updated after the patch
has stopped using interval?
5. How does this feature interacts with the SKIP feature? Currently,
it doesn't care whether the changes of a particular xact are skipped
or not. I think that might be okay because anyway the purpose of this
feature is to make subscriber lag from publishers. What do you think?
I feel we can add some comments to indicate the same.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-23 22:15 Peter Smith <[email protected]>
parent: Amit Kapila <[email protected]>
1 sibling, 0 replies; 40+ messages in thread
From: Peter Smith @ 2023-01-23 22:15 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Takamichi Osumi (Fujitsu) <[email protected]>; vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Mon, Jan 23, 2023 at 9:44 PM Amit Kapila <[email protected]> wrote:
>
> On Mon, Jan 23, 2023 at 1:36 PM Peter Smith <[email protected]> wrote:
> >
> > Here are my review comments for v19-0001.
> >
> ...
> >
> > 5. parse_subscription_options
> >
> > + /*
> > + * The combination of parallel streaming mode and min_apply_delay is not
> > + * allowed. The subscriber in the parallel streaming mode applies each
> > + * stream on arrival without the time of commit/prepare. So, the
> > + * subscriber needs to depend on the arrival time of the stream in this
> > + * case, if we apply the time-delayed feature for such transactions. Then
> > + * there is a possibility where some unnecessary delay will be added on
> > + * the subscriber by network communication break between nodes or other
> > + * heavy work load on the publisher. On the other hand, applying the delay
> > + * at the end of transaction with parallel apply also can cause issues of
> > + * used resource bloat and locks kept in open for a long time. Thus, those
> > + * features can't work together.
> > + */
> >
> > IMO some re-wording might be warranted here. I am not sure quite how
> > to do it. Perhaps like below?
> >
> > SUGGESTION
> >
> > The combination of parallel streaming mode and min_apply_delay is not allowed.
> >
> > Here are some reasons why these features are incompatible:
> > a. In the parallel streaming mode the subscriber applies each stream
> > on arrival without knowledge of the commit/prepare time. This means we
> > cannot calculate the underlying network/decoding lag between publisher
> > and subscriber, and so always waiting for the full 'min_apply_delay'
> > period might include unnecessary delay.
> > b. If we apply the delay at the end of the transaction of the parallel
> > apply then that would cause issues related to resource bloat and locks
> > being held for a long time.
> >
> > ~~~
> >
>
> How about something like:
> The combination of parallel streaming mode and min_apply_delay is not
> allowed. This is because we start applying the transaction stream as
> soon as the first change arrives without knowing the transaction's
> prepare/commit time. This means we cannot calculate the underlying
> network/decoding lag between publisher and subscriber, and so always
> waiting for the full 'min_apply_delay' period might include
> unnecessary delay.
>
> The other possibility is to apply the delay at the end of the parallel
> apply transaction but that would cause issues related to resource
> bloat and locks being held for a long time.
>
+1. That's better.
>
> > 6. defGetMinApplyDelay
> >
...
> >
> > 6b.
> > I thought this function should be implemented as static and located at
> > the top of the subscriptioncmds.c source file.
> >
>
> I agree that this should be a static function but I think its current
> location is a better place as other similar function is just above it.
>
But, why not do everything, instead of settling on a half-fix?
e.g.
1. Change the new function (defGetMinApplyDelay) to be static as it should be
2. And move defGetMinApplyDelay to the top of the file where IMO it
really belongs
3. And then remove the (now) redundant forward declaration of
defGetMinApplyDelay
4. And also move the existing function (defGetStreamingMode) to the
top of the file so that those similar functions (defGetMinApplyDelay
and defGetStreamingMode) can remain together
------
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-23 23:32 Euler Taveira <[email protected]>
parent: Takamichi Osumi (Fujitsu) <[email protected]>
2 siblings, 3 replies; 40+ messages in thread
From: Euler Taveira @ 2023-01-23 23:32 UTC (permalink / raw)
To: [email protected] <[email protected]>; Peter Smith <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Dilip Kumar <[email protected]>; Amit Kapila <[email protected]>; Melih Mutlu <[email protected]>; Andres Freund <[email protected]>; Marcos Pegoraro <[email protected]>; pgsql-hackers
On Sun, Jan 22, 2023, at 9:42 AM, Takamichi Osumi (Fujitsu) wrote:
> On Saturday, January 21, 2023 3:36 AM I wrote:
> > Kindly have a look at the patch v18.
> I've conducted some refactoring for v18.
> Now the latest patch should be tidier and
> the comments would be clearer and more aligned as a whole.
>
> Attached the updated patch v19.
[I haven't been following this thread for a long time...]
Good to know that you keep improving this patch. I have a few suggestions that
were easier to provide a patch on top of your latest patch than to provide an
inline suggestions.
There are a few documentation polishing. Let me comment some of them above.
- The length of time (ms) to delay the application of changes.
+ Total time spent delaying the application of changes, in milliseconds
I don't remember if I suggested this description for catalog but IMO the
suggestion reads better for me.
- For time-delayed logical replication (i.e. when the subscription is
- created with parameter min_apply_delay > 0), the apply worker sends a
- Standby Status Update message to the publisher with a period of
- <literal>wal_receiver_status_interval</literal>. Make sure to set
- <literal>wal_receiver_status_interval</literal> less than the
- <literal>wal_sender_timeout</literal> on the publisher, otherwise, the
- walsender will repeatedly terminate due to the timeout errors. If
- <literal>wal_receiver_status_interval</literal> is set to zero, the apply
- worker doesn't send any feedback messages during the subscriber's
- <literal>min_apply_delay</literal> period. See
- <xref linkend="sql-createsubscription"/> for details.
+ For time-delayed logical replication, the apply worker sends a feedback
+ message to the publisher every
+ <varname>wal_receiver_status_interval</varname> milliseconds. Make sure
+ to set <varname>wal_receiver_status_interval</varname> less than the
+ <varname>wal_sender_timeout</varname> on the publisher, otherwise, the
+ <literal>walsender</literal> will repeatedly terminate due to timeout
+ error. If <varname>wal_receiver_status_interval</varname> is set to
+ zero, the apply worker doesn't send any feedback messages during the
+ <literal>min_apply_delay</literal> interval.
I removed the parenthesis explanation about time-delayed logical replication.
If you are reading the documentation and does not know what it means you should
(a) read the logical replication chapter or (b) check the glossary (maybe a new
entry should be added). I also removed the Standby status Update message but it
is a low level detail; let's refer to it as feedback message as the other
sentences do. I changed "literal" to "varname" that's the correct tag for
parameters. I replace "period" with "interval" that was the previous
terminology. IMO we should be uniform, use one or the other.
- The subscriber replication can be instructed to lag behind the publisher
- side changes by specifying the <literal>min_apply_delay</literal>
- subscription parameter. See <xref linkend="sql-createsubscription"/> for
- details.
+ A logical replication subscription can delay the application of changes by
+ specifying the <literal>min_apply_delay</literal> subscription parameter.
+ See <xref linkend="sql-createsubscription"/> for details.
This feature refers to a specific subscription, hence, "logical replication
subscription" instead of "subscriber replication".
+ if (IsSet(opts->specified_opts, SUBOPT_MIN_APPLY_DELAY))
+ errorConflictingDefElem(defel, pstate);
+
Peter S referred to this missing piece of code too.
-int
+static int
defGetMinApplyDelay(DefElem *def)
{
It seems you forgot static keyword.
- elog(DEBUG2, "time-delayed replication for txid %u, min_apply_delay = %lld ms, Remaining wait time: %ld ms",
- xid, (long long) MySubscription->minapplydelay, diffms);
+ elog(DEBUG2, "time-delayed replication for txid %u, min_apply_delay = " INT64_FORMAT " ms, remaining wait time: %ld ms",
+ xid, MySubscription->minapplydelay, diffms);
int64 should use format modifier INT64_FORMAT.
- (long) wal_receiver_status_interval * 1000,
+ wal_receiver_status_interval * 1000L,
Cast is not required. I added a suffix to the constant.
- elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write %X/%X, flush %X/%X in-delayed: %d",
+ elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write %X/%X, flush %X/%X, apply delay: %s",
force,
LSN_FORMAT_ARGS(recvpos),
LSN_FORMAT_ARGS(writepos),
LSN_FORMAT_ARGS(flushpos),
- in_delayed_apply);
+ in_delayed_apply? "yes" : "no");
It is better to use a string to represent the yes/no option.
- gettext_noop("Min apply delay (ms)"));
+ gettext_noop("Min apply delay"));
I don't know if it was discussed but we don't add units to headers. When I
think about this parameter representation (internal and external), I decided to
use the previous code because it provides a unit for external representation. I
understand that using the same representation as recovery_min_apply_delay is
good but the current code does not handle the external representation
accordingly. (recovery_min_apply_delay uses the GUC machinery to adds the unit
but for min_apply_delay, it doesn't).
# Setup for streaming case
-$node_publisher->append_conf('postgres.conf',
+$node_publisher->append_conf('postgresql.conf',
'logical_decoding_mode = immediate');
$node_publisher->reload;
Fix configuration file name.
Maybe tests should do a better job. I think check_apply_delay_time is fragile
because it does not guarantee that time is not shifted. Time-delayed
replication is a subscriber feature and to check its correctness it should
check the logs.
# Note that we cannot call check_apply_delay_log() here because there is a
# possibility that the delay is skipped. The event happens when the WAL
# replication between publisher and subscriber is delayed due to a mechanical
# problem. The log output will be checked later - substantial delay-time case.
If you might not use the logs for it, it should adjust the min_apply_delay, no?
It does not exercise the min_apply_delay vs parallel streaming mode.
+ /*
+ * The combination of parallel streaming mode and
+ * min_apply_delay is not allowed.
+ */
+ if (opts.streaming == LOGICALREP_STREAM_PARALLEL)
+ if ((IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) && opts.min_apply_delay > 0) ||
+ (!IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) && sub->minapplydelay > 0))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable %s mode for subscription with %s",
+ "streaming = parallel", "min_apply_delay"));
+
Is this code correct? I also didn't like this message. "cannot enable streaming
= parallel mode for subscription with min_apply_delay" is far from a good error
message. How about refer parallelism to "parallel streaming mode".
--
Euler Taveira
EDB https://www.enterprisedb.com/
Attachments:
[text/x-patch] review-1.patch (49.9K, ../../[email protected]/3-review-1.patch)
download | inline diff:
From 5024325284ee3b4a4dc0a6a1cc6457ed5608cb46 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 23 Jan 2023 15:52:55 -0300
Subject: [PATCH] Euler's review
---
doc/src/sgml/catalogs.sgml | 2 +-
doc/src/sgml/config.sgml | 20 ++-
doc/src/sgml/logical-replication.sgml | 7 +-
doc/src/sgml/ref/create_subscription.sgml | 13 +-
src/backend/commands/subscriptioncmds.c | 13 +-
src/backend/replication/logical/worker.c | 40 +++---
src/bin/psql/describe.c | 2 +-
src/test/regress/expected/subscription.out | 160 ++++++++++-----------
src/test/subscription/t/032_apply_delay.pl | 8 +-
9 files changed, 133 insertions(+), 132 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index bf3c05241c..0bdb683296 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7878,7 +7878,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<structfield>subminapplydelay</structfield> <type>int8</type>
</para>
<para>
- The length of time (ms) to delay the application of changes.
+ Total time spent delaying the application of changes, in milliseconds
</para></entry>
</row>
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 39244bf64a..a15723d74f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4788,17 +4788,15 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
command line.
</para>
<para>
- For time-delayed logical replication (i.e. when the subscription is
- created with parameter min_apply_delay > 0), the apply worker sends a
- Standby Status Update message to the publisher with a period of
- <literal>wal_receiver_status_interval</literal>. Make sure to set
- <literal>wal_receiver_status_interval</literal> less than the
- <literal>wal_sender_timeout</literal> on the publisher, otherwise, the
- walsender will repeatedly terminate due to the timeout errors. If
- <literal>wal_receiver_status_interval</literal> is set to zero, the apply
- worker doesn't send any feedback messages during the subscriber's
- <literal>min_apply_delay</literal> period. See
- <xref linkend="sql-createsubscription"/> for details.
+ For time-delayed logical replication, the apply worker sends a feedback
+ message to the publisher every
+ <varname>wal_receiver_status_interval</varname> milliseconds. Make sure
+ to set <varname>wal_receiver_status_interval</varname> less than the
+ <varname>wal_sender_timeout</varname> on the publisher, otherwise, the
+ <literal>walsender</literal> will repeatedly terminate due to timeout
+ error. If <varname>wal_receiver_status_interval</varname> is set to
+ zero, the apply worker doesn't send any feedback messages during the
+ <literal>min_apply_delay</literal> interval.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 863af11a47..d8ae93f88d 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -248,10 +248,9 @@
</para>
<para>
- The subscriber replication can be instructed to lag behind the publisher
- side changes by specifying the <literal>min_apply_delay</literal>
- subscription parameter. See <xref linkend="sql-createsubscription"/> for
- details.
+ A logical replication subscription can delay the application of changes by
+ specifying the <literal>min_apply_delay</literal> subscription parameter.
+ See <xref linkend="sql-createsubscription"/> for details.
</para>
<sect2 id="logical-replication-subscription-slot">
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 76ee9c0b3d..97ca9f8d9e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -378,10 +378,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
<warning>
<para>
- Delaying the replication can mean there is a much longer time between making
- a change on the publisher, and that change being committed on the subscriber.
- This can impact the performance of synchronous replication.
- See <xref linkend="guc-synchronous-commit"/>.
+ Delaying the replication can mean there is a much longer time
+ between making a change on the publisher, and that change being
+ committed on the subscriber. This can impact the performance of
+ synchronous replication. See <xref linkend="guc-synchronous-commit"/>
+ parameter.
</para>
</warning>
</listitem>
@@ -452,8 +453,8 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
<para>
- A non-zero <literal>min_apply_delay</literal> parameter is not allowed when streaming
- in parallel mode.
+ A non-zero <literal>min_apply_delay</literal> parameter is not allowed when
+ streaming in parallel mode.
</para>
<para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 11e9e9160a..d5fa7a95a9 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -331,6 +331,9 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
else if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
strcmp(defel->defname, "min_apply_delay") == 0)
{
+ if (IsSet(opts->specified_opts, SUBOPT_MIN_APPLY_DELAY))
+ errorConflictingDefElem(defel, pstate);
+
opts->specified_opts |= SUBOPT_MIN_APPLY_DELAY;
opts->min_apply_delay = defGetMinApplyDelay(defel);
}
@@ -2261,11 +2264,11 @@ defGetStreamingMode(DefElem *def)
/*
- * Extract the min_apply_delay mode value from a DefElem. This is very similar
- * to PGC_INT case of parse_and_validate_value(), because min_apply_delay
+ * Extract the min_apply_delay value from a DefElem. This is very similar to
+ * parse_and_validate_value() for integer values, because min_apply_delay
* accepts the same string as recovery_min_apply_delay.
*/
-int
+static int
defGetMinApplyDelay(DefElem *def)
{
char *value;
@@ -2294,8 +2297,8 @@ defGetMinApplyDelay(DefElem *def)
hintmsg ? errhint("%s", _(hintmsg)) : 0));
/*
- * Check lower bound. parse_int() has been already confirmed that result
- * is equal to or smaller than INT_MAX.
+ * Check lower bound. parse_int() has already been confirmed that result
+ * is less than or equal to INT_MAX.
*/
if (result < 0)
ereport(ERROR,
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index eeac69ea13..00fe29fc20 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -320,11 +320,11 @@ bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
/*
- * In order to avoid walsender timeout for time-delayed replication the worker
- * process keeps sending feedback messages during the delay period.
- * Meanwhile, the feature delays the apply before starting the
- * transaction and thus we don't write WALs for the suspended changes during
- * the wait. When the worker process sends a feedback message
+ * In order to avoid walsender timeout for time-delayed logical replication the
+ * apply worker keeps sending feedback messages during the delay interval.
+ * Meanwhile, the feature delays the apply before the start of the
+ * transaction and thus we don't write WAL records for the suspended changes during
+ * the wait. When the apply worker sends a feedback message
* during the delay, we should not make positions of the flushed and apply LSN
* overwritten by the last received latest LSN. See send_feedback() for details.
*/
@@ -1090,20 +1090,20 @@ maybe_delay_apply(TransactionId xid, TimestampTz finish_ts)
if (diffms <= 0)
break;
- elog(DEBUG2, "time-delayed replication for txid %u, min_apply_delay = %lld ms, Remaining wait time: %ld ms",
- xid, (long long) MySubscription->minapplydelay, diffms);
+ elog(DEBUG2, "time-delayed replication for txid %u, min_apply_delay = " INT64_FORMAT " ms, remaining wait time: %ld ms",
+ xid, MySubscription->minapplydelay, diffms);
/*
* Call send_feedback() to prevent the publisher from exiting by
* timeout during the delay, when wal_receiver_status_interval is
* available.
*/
- if (wal_receiver_status_interval > 0
- && diffms > wal_receiver_status_interval * 1000)
+ if (wal_receiver_status_interval > 0 &&
+ diffms > wal_receiver_status_interval * 1000L)
{
WaitLatch(MyLatch,
WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
- (long) wal_receiver_status_interval * 1000,
+ wal_receiver_status_interval * 1000L,
WAIT_EVENT_RECOVERY_APPLY_DELAY);
send_feedback(last_received, true, false, true);
}
@@ -2135,8 +2135,8 @@ ensure_last_message(FileSet *stream_fileset, TransactionId xid, int fileno,
/*
* Common spoolfile processing.
*
- * The commit/prepare time for streaming transaction is required to achieve
- * time-delayed replication.
+ * The commit/prepare time (finish_ts) for streamed transactions is required
+ * for time-delayed logical replication.
*/
void
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
@@ -3869,12 +3869,12 @@ send_feedback(XLogRecPtr recvpos, bool force, bool requestReply, bool in_delayed
* No outstanding transactions to flush, we can report the latest received
* position. This is important for synchronous replication.
*
- * If the subscriber side apply is delayed (because of time-delayed
- * replication) then do not tell the publisher that the received latest
- * LSN is already applied and flushed, otherwise, it leads to the
- * publisher side making a wrong assumption of logical replication
- * progress. Instead, we just send a feedback message to avoid a publisher
- * timeout during the delay.
+ * If the logical replication subscription is delayed (min_apply_delay
+ * parameter) then do not inform the publisher that the received latest LSN
+ * is already applied and flushed, otherwise, the publisher will make a
+ * wrong assumption about the logical replication progress. Instead, it
+ * just sends a feedback message to avoid a replication timeout during the
+ * delay.
*/
if (!have_pending_txes && !in_delayed_apply)
flushpos = writepos = recvpos;
@@ -3913,12 +3913,12 @@ send_feedback(XLogRecPtr recvpos, bool force, bool requestReply, bool in_delayed
pq_sendint64(reply_message, now); /* sendTime */
pq_sendbyte(reply_message, requestReply); /* replyRequested */
- elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write %X/%X, flush %X/%X in-delayed: %d",
+ elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write %X/%X, flush %X/%X, apply delay: %s",
force,
LSN_FORMAT_ARGS(recvpos),
LSN_FORMAT_ARGS(writepos),
LSN_FORMAT_ARGS(flushpos),
- in_delayed_apply);
+ in_delayed_apply? "yes" : "no");
walrcv_send(LogRepWorkerWalRcvConn,
reply_message->data, reply_message->len);
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 8a27063bed..81d4607a1c 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6533,7 +6533,7 @@ describeSubscriptions(const char *pattern, bool verbose)
", suborigin AS \"%s\"\n"
", subminapplydelay AS \"%s\"\n",
gettext_noop("Origin"),
- gettext_noop("Min apply delay (ms)"));
+ gettext_noop("Min apply delay"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 1230bcb096..977f73fe9b 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -114,18 +114,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -135,10 +135,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -155,10 +155,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -167,10 +167,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -202,10 +202,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | 0 | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -239,19 +239,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -263,27 +263,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -298,10 +298,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -316,10 +316,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -355,10 +355,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -367,10 +367,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -380,10 +380,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,18 +396,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | 0 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | 0 | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -425,19 +425,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 123 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 123 | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- success -- min_apply_delay value with unit is converted into ms and stored as an integer
ALTER SUBSCRIPTION regress_testsub SET (min_apply_delay = '1 d');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay (ms) | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+----------------------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 86400000 | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Min apply delay | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | 86400000 | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - alter subscription with streaming = parallel should fail when time-delayed replication is set
diff --git a/src/test/subscription/t/032_apply_delay.pl b/src/test/subscription/t/032_apply_delay.pl
index 37388e474f..1b6bc1ef80 100644
--- a/src/test/subscription/t/032_apply_delay.pl
+++ b/src/test/subscription/t/032_apply_delay.pl
@@ -16,7 +16,7 @@ sub check_apply_delay_log
{
my ($node_subscriber, $offset, $expected) = @_;
- my $log_location = $node_subscriber->wait_for_log(qr/time-delayed replication for txid (\d+), min_apply_delay = (\d+) ms, Remaining wait time: (\d+) ms/, $offset);
+ my $log_location = $node_subscriber->wait_for_log(qr/time-delayed replication for txid (\d+), min_apply_delay = (\d+) ms, remaining wait time: (\d+) ms/, $offset);
cmp_ok($log_location, '>', $offset,
"logfile contains triggered logical replication apply delay"
@@ -25,7 +25,7 @@ sub check_apply_delay_log
# Get the delay time from the server log
my $contents = slurp_file($node_subscriber->logfile, $offset);
$contents =~
- qr/time-delayed replication for txid (\d+), min_apply_delay = (\d+) ms, Remaining wait time: (\d+) ms/
+ qr/time-delayed replication for txid (\d+), min_apply_delay = (\d+) ms, remaining wait time: (\d+) ms/
or die "could not get the apply worker wait time";
my $logged_delay = $3;
@@ -87,7 +87,7 @@ $node_publisher->safe_psql('postgres',
my $appname = 'tap_sub';
-# Create a subscription that applies the trasaction after 50 milliseconds delay
+# Create a subscription that applies the transaction after 50 milliseconds delay
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (copy_data = off, min_apply_delay = '50ms', streaming = 'on')"
);
@@ -114,7 +114,7 @@ is($result, qq(2|1|2), 'check if the new rows were applied to subscriber');
check_apply_delay_time($node_publisher, $node_subscriber, '2', '0.05');
# Setup for streaming case
-$node_publisher->append_conf('postgres.conf',
+$node_publisher->append_conf('postgresql.conf',
'logical_decoding_mode = immediate');
$node_publisher->reload;
--
2.30.2
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-24 00:45 Kyotaro Horiguchi <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Kyotaro Horiguchi @ 2023-01-24 00:45 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
At Mon, 23 Jan 2023 17:36:13 +0530, Amit Kapila <[email protected]> wrote in
> On Sun, Jan 22, 2023 at 6:12 PM Takamichi Osumi (Fujitsu)
> <[email protected]> wrote:
> >
> >
> > Attached the updated patch v19.
> Few comments:
> 2.
> + if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
> + opts->min_apply_delay > 0 && opts->streaming == LOGICALREP_STREAM_PARALLEL)
> + ereport(ERROR,
> + errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("%s and %s are mutually exclusive options",
> + "min_apply_delay > 0", "streaming = parallel"));
> }
>
> I think here we should add a comment for the translator as we are
> doing in some other nearby cases.
IMHO "foo > bar" is not an "option". I think we say "foo and bar are
mutually exclusive options" but I think don't say "foo = x and bar = y
are.. options". I wrote a comment as "this should be more like
human-speaking" and Euler seems having the same feeling for another
error message.
Concretely I would spell this as "min_apply_delay cannot be enabled
when parallel streaming mode is enabled" or something. And the
opposite-direction message nearby would be "parallel streaming mode
cannot be enabled when min_apply_delay is enabled."
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-24 05:18 Amit Kapila <[email protected]>
parent: Euler Taveira <[email protected]>
2 siblings, 0 replies; 40+ messages in thread
From: Amit Kapila @ 2023-01-24 05:18 UTC (permalink / raw)
To: Euler Taveira <[email protected]>; +Cc: [email protected] <[email protected]>; Peter Smith <[email protected]>; vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Dilip Kumar <[email protected]>; Melih Mutlu <[email protected]>; Andres Freund <[email protected]>; Marcos Pegoraro <[email protected]>; pgsql-hackers
On Tue, Jan 24, 2023 at 5:02 AM Euler Taveira <[email protected]> wrote:
>
> On Sun, Jan 22, 2023, at 9:42 AM, Takamichi Osumi (Fujitsu) wrote:
>
>
> Attached the updated patch v19.
>
> [I haven't been following this thread for a long time...]
>
> Good to know that you keep improving this patch. I have a few suggestions that
> were easier to provide a patch on top of your latest patch than to provide an
> inline suggestions.
>
Euler, thanks for your comments. We have an existing problem related
to shutdown which impacts this patch. The problem is that during
shutdown on the publisher, we wait for all the WAL to be sent and
flushed on the subscriber. Now, if we user has configured a long value
for min_apply_delay on the subscriber then the shutdown won't be
successful. This can happen even today if the subscriber waits for
some lock during the apply. This is not so much a problem with
physical replication because there we have a separate process to first
flush the WAL. This problem has been discussed in a separate thread as
well. See [1]. It is important to reach conclusion even if we just
want to document it. So, your thoughts on that other thread can help
us to make it move forward.
[1] - https://www.postgresql.org/message-id/TYAPR01MB586668E50FC2447AD7F92491F5E89%40TYAPR01MB5866.jpnprd0...
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-24 14:03 Takamichi Osumi (Fujitsu) <[email protected]>
parent: Amit Kapila <[email protected]>
1 sibling, 0 replies; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-24 14:03 UTC (permalink / raw)
To: 'Amit Kapila' <[email protected]>; Peter Smith <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Monday, January 23, 2023 7:45 PM Amit Kapila <[email protected]> wrote:
> On Mon, Jan 23, 2023 at 1:36 PM Peter Smith <[email protected]>
> wrote:
> >
> > Here are my review comments for v19-0001.
> >
> ...
> >
> > 5. parse_subscription_options
> >
> > + /*
> > + * The combination of parallel streaming mode and min_apply_delay is
> > + not
> > + * allowed. The subscriber in the parallel streaming mode applies
> > + each
> > + * stream on arrival without the time of commit/prepare. So, the
> > + * subscriber needs to depend on the arrival time of the stream in
> > + this
> > + * case, if we apply the time-delayed feature for such transactions.
> > + Then
> > + * there is a possibility where some unnecessary delay will be added
> > + on
> > + * the subscriber by network communication break between nodes or
> > + other
> > + * heavy work load on the publisher. On the other hand, applying the
> > + delay
> > + * at the end of transaction with parallel apply also can cause
> > + issues of
> > + * used resource bloat and locks kept in open for a long time. Thus,
> > + those
> > + * features can't work together.
> > + */
> >
> > IMO some re-wording might be warranted here. I am not sure quite how
> > to do it. Perhaps like below?
> >
> > SUGGESTION
> >
> > The combination of parallel streaming mode and min_apply_delay is not
> allowed.
> >
> > Here are some reasons why these features are incompatible:
> > a. In the parallel streaming mode the subscriber applies each stream
> > on arrival without knowledge of the commit/prepare time. This means we
> > cannot calculate the underlying network/decoding lag between publisher
> > and subscriber, and so always waiting for the full 'min_apply_delay'
> > period might include unnecessary delay.
> > b. If we apply the delay at the end of the transaction of the parallel
> > apply then that would cause issues related to resource bloat and locks
> > being held for a long time.
> >
> > ~~~
> >
>
> How about something like:
> The combination of parallel streaming mode and min_apply_delay is not
> allowed. This is because we start applying the transaction stream as soon as
> the first change arrives without knowing the transaction's prepare/commit time.
> This means we cannot calculate the underlying network/decoding lag between
> publisher and subscriber, and so always waiting for the full 'min_apply_delay'
> period might include unnecessary delay.
>
> The other possibility is to apply the delay at the end of the parallel apply
> transaction but that would cause issues related to resource bloat and locks
> being held for a long time.
Thank you for providing a good description ! Adopted.
The latest patch can be seen in [1].
[1] - https://www.postgresql.org/message-id/TYCPR01MB8373DC1881F382B4703F26E0EDC99%40TYCPR01MB8373.jpnprd0...
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-24 14:22 Takamichi Osumi (Fujitsu) <[email protected]>
parent: Peter Smith <[email protected]>
1 sibling, 0 replies; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-24 14:22 UTC (permalink / raw)
To: 'Peter Smith' <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Monday, January 23, 2023 5:07 PM Peter Smith <[email protected]> wrote:
> Here are my review comments for v19-0001.
Thanks for your review !
>
> ======
> Commit message
>
> 1.
> The combination of parallel streaming mode and min_apply_delay is not
> allowed. The subscriber in the parallel streaming mode applies each stream on
> arrival without the time of commit/prepare. So, the subscriber needs to depend
> on the arrival time of the stream in this case, if we apply the time-delayed
> feature for such transactions. Then there is a possibility where some
> unnecessary delay will be added on the subscriber by network communication
> break between nodes or other heavy work load on the publisher. On the other
> hand, applying the delay at the end of transaction with parallel apply also can
> cause issues of used resource bloat and locks kept in open for a long time.
> Thus, those features can't work together.
> ~
>
> I think the above is just cut/paste from a code comment within
> subscriptioncmds.c. See review comments #5 below -- so if the code is
> changed then this commit message should also change to match it.
Now, updated this. Kindly have a look at the latest patch in [1].
>
> ======
> doc/src/sgml/ref/create_subscription.sgml
>
> 2.
> + <varlistentry>
> + <term><literal>min_apply_delay</literal>
> (<type>integer</type>)</term>
> + <listitem>
> + <para>
> + By default, the subscriber applies changes as soon as possible.
> This
> + parameter allows the user to delay the application of changes by a
> + given time interval. If the value is specified without units, it is
> + taken as milliseconds. The default is zero (no delay).
> + </para>
>
> 2a.
> The pgdocs says this is an integer default to “ms” unit. Also, the example on
> this same page shows it is set to '4h'. But I did not see any mention of what
> other units are available to the user. Maybe other time units should be
> mentioned here, or maybe a link should be given to the section “20.1.1.
> Parameter Names and Values".
Added.
> ~
>
> 2b.
> Previously the word "interval" was deliberately used because this parameter
> had interval support. But maybe now it should be changed so it is not
> misleading.
>
> "a given time interval" --> "a given time period" ??
Fixed.
> ======
> src/backend/commands/subscriptioncmds.c
>
> 3. Forward declare
>
> +static int defGetMinApplyDelay(DefElem *def);
>
> If the new function is implemented as static near the top of this source file then
> this forward declare would not even be necessary, right?
This declaration has been kept as discussed.
> ~~~
>
> 4. parse_subscription_options
>
> @@ -324,6 +328,12 @@ parse_subscription_options(ParseState *pstate, List
> *stmt_options,
> opts->specified_opts |= SUBOPT_LSN;
> opts->lsn = lsn;
> }
> + else if (IsSet(supported_opts, SUBOPT_MIN_APPLY_DELAY) &&
> + strcmp(defel->defname, "min_apply_delay") == 0) {
> + opts->specified_opts |= SUBOPT_MIN_APPLY_DELAY; min_apply_delay =
> + opts->defGetMinApplyDelay(defel);
> + }
>
> Should this code fragment be calling errorConflictingDefElem so it will report
> an error if the same min_apply_delay parameter is redundantly repeated?
> (IIUC, this appears to be the code pattern for other parameters nearby).
Added.
> ~~~
>
> 5. parse_subscription_options
>
> + /*
> + * The combination of parallel streaming mode and min_apply_delay is
> + not
> + * allowed. The subscriber in the parallel streaming mode applies each
> + * stream on arrival without the time of commit/prepare. So, the
> + * subscriber needs to depend on the arrival time of the stream in this
> + * case, if we apply the time-delayed feature for such transactions.
> + Then
> + * there is a possibility where some unnecessary delay will be added on
> + * the subscriber by network communication break between nodes or other
> + * heavy work load on the publisher. On the other hand, applying the
> + delay
> + * at the end of transaction with parallel apply also can cause issues
> + of
> + * used resource bloat and locks kept in open for a long time. Thus,
> + those
> + * features can't work together.
> + */
>
> IMO some re-wording might be warranted here. I am not sure quite how to do it.
> Perhaps like below?
>
> SUGGESTION
>
> The combination of parallel streaming mode and min_apply_delay is not
> allowed.
>
> Here are some reasons why these features are incompatible:
> a. In the parallel streaming mode the subscriber applies each stream on arrival
> without knowledge of the commit/prepare time. This means we cannot
> calculate the underlying network/decoding lag between publisher and
> subscriber, and so always waiting for the full 'min_apply_delay'
> period might include unnecessary delay.
> b. If we apply the delay at the end of the transaction of the parallel apply then
> that would cause issues related to resource bloat and locks being held for a
> long time.
Now, this has been changed to the one suggested by Amit-san.
Thanks for your help.
> ~~~
>
> 6. defGetMinApplyDelay
>
> +
> +
> +/*
> + * Extract the min_apply_delay mode value from a DefElem. This is very
> +similar
> + * to PGC_INT case of parse_and_validate_value(), because
> +min_apply_delay
> + * accepts the same string as recovery_min_apply_delay.
> + */
> +int
> +defGetMinApplyDelay(DefElem *def)
>
> 6a.
> "same string" -> "same parameter format" ??
Fixed.
> ~
>
> 6b.
> I thought this function should be implemented as static and located at the top
> of the subscriptioncmds.c source file.
Made it static but didn't change the place, as Amit-san mentioned.
> ======
> src/backend/replication/logical/worker.c
>
> 7. maybe_delay_apply
>
> +static void maybe_delay_apply(TransactionId xid, TimestampTz
> +finish_ts);
>
> Is there a reason why this is here? AFAIK the static implementation precedes
> any usage so I doubt this forward declaration is required.
Removed.
> ~~~
>
> 8. send_feedback
>
> @@ -3775,11 +3912,12 @@ send_feedback(XLogRecPtr recvpos, bool force,
> bool requestReply)
> pq_sendint64(reply_message, now); /* sendTime */
> pq_sendbyte(reply_message, requestReply); /* replyRequested */
>
> - elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write %X/%X,
> flush %X/%X",
> + elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write
> %X/%X, flush %X/%X in-delayed: %d",
> force,
> LSN_FORMAT_ARGS(recvpos),
> LSN_FORMAT_ARGS(writepos),
> - LSN_FORMAT_ARGS(flushpos));
> + LSN_FORMAT_ARGS(flushpos),
> + in_delayed_apply);
>
> Wondering if it is better to write this as:
> "sending feedback (force %d, in_delayed_apply %d) to recv %X/%X,
> write %X/%X, flush %X/%X"
Adopted and merged with the modification Euler-san provided.
> ~
>
> 10. Add new tests?
>
> Should there be other tests just to verify different units (like 'd', 'h', 'min') are
> working OK?
No need. The current subscription.sql does the check
of "invalid value for parameter..." error message, which ensures we call
the defGetMinApplyDelay(). Additionally, we have the test of one unit 'd'
for unit iteration loopin convert_to_base_unit().
So, the current test sets should suffice.
> ======
> src/test/subscription/t/032_apply_delay.pl
>
> 11.
> +# Confirm the time-delayed replication has been effective from the
> +server log # message where the apply worker emits for applying delay.
> +Moreover, verifies # that the current worker's delayed time is
> +sufficiently bigger than the # expected value, in order to check any update of
> the min_apply_delay.
> +sub check_apply_delay_log
>
> "the current worker's delayed time..." --> "the current worker's remaining wait
> time..." ??
Fixed.
> ~~~
>
> 12.
> + # Get the delay time from the server log my $contents =
> + slurp_file($node_subscriber->logfile, $offset);
>
> "Get the delay time...." --> "Get the remaining wait time..."
Fixed.
> ~~~
>
> 13.
> +# Create a subscription that applies the trasaction after 50
> +milliseconds delay $node_subscriber->safe_psql('postgres',
> + "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr
> application_name=$appname' PUBLICATION tap_pub WITH (copy_data = off,
> min_apply_delay = '50ms', streaming = 'on')"
> +);
>
> 13a.
> typo: "trasaction"
Fixed.
> ~
>
> 13b
> 50ms seems an extremely short time – How do you even know if this is testing
> anything related to the time delay? You may just be detecting the normal lag
> between publisher and subscriber without time delay having much to do with
> anything.
The wait time has been updated to 1 second now.
Also, the TAP tests now search for the emitted logs by the apply worker.
The path to emit the log is in the maybe_apply_delay and
it does writes the log only if the "diffms" is bigger than zero,
which invokes the wait. So, this will ensure we use the feature
by this flow.
> ~
>
> 14.
>
> +# Note that we cannot call check_apply_delay_log() here because there
> +is a # possibility that the delay is skipped. The event happens when
> +the WAL # replication between publisher and subscriber is delayed due
> +to a mechanical # problem. The log output will be checked later - substantial
> delay-time case.
> +
> +# Verify that the subscriber lags the publisher by at least 50
> +milliseconds check_apply_delay_time($node_publisher, $node_subscriber,
> +'2', '0.05');
>
> 14a.
> "The event happens..." ??
>
> Did you mean "This might happen if the WAL..."
This part has been removed.
> ~
>
> 14b.
> The log output will be checked later - substantial delay-time case.
>
> I think that needs re-wording to clarify.
> e.g1. you have nothing called a "substantial delay-time" case.
> e.g2. the word "later" confused me. Originally, I thought you meant it is not
> tested yet but that you will check it "later", but now IIUC you are just referring
> to the "1 day 5 minutes" test that comes below in this location TAP file (??)
Also, removed.
[1] - https://www.postgresql.org/message-id/TYCPR01MB8373DC1881F382B4703F26E0EDC99%40TYCPR01MB8373.jpnprd0...
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-01-24 14:57 Takamichi Osumi (Fujitsu) <[email protected]>
parent: Euler Taveira <[email protected]>
2 siblings, 0 replies; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-01-24 14:57 UTC (permalink / raw)
To: 'Euler Taveira' <[email protected]>; Peter Smith <[email protected]>; +Cc: vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; Dilip Kumar <[email protected]>; Amit Kapila <[email protected]>; Melih Mutlu <[email protected]>; Andres Freund <[email protected]>; Marcos Pegoraro <[email protected]>; pgsql-hackers
On Tuesday, January 24, 2023 8:32 AM Euler Taveira <[email protected]> wrote:
> Good to know that you keep improving this patch. I have a few suggestions that
> were easier to provide a patch on top of your latest patch than to provide an
> inline suggestions.
Thanks for your review ! We basically adopted your suggestions.
> There are a few documentation polishing. Let me comment some of them above.
>
> - The length of time (ms) to delay the application of changes.
> + Total time spent delaying the application of changes, in milliseconds
>
> I don't remember if I suggested this description for catalog but IMO the
> suggestion reads better for me.
Adopted the above change.
> - For time-delayed logical replication (i.e. when the subscription is
> - created with parameter min_apply_delay > 0), the apply worker sends a
> - Standby Status Update message to the publisher with a period of
> - <literal>wal_receiver_status_interval</literal>. Make sure to set
> - <literal>wal_receiver_status_interval</literal> less than the
> - <literal>wal_sender_timeout</literal> on the publisher, otherwise, the
> - walsender will repeatedly terminate due to the timeout errors. If
> - <literal>wal_receiver_status_interval</literal> is set to zero, the apply
> - worker doesn't send any feedback messages during the subscriber's
> - <literal>min_apply_delay</literal> period. See
> - <xref linkend="sql-createsubscription"/> for details.
> + For time-delayed logical replication, the apply worker sends a feedback
> + message to the publisher every
> + <varname>wal_receiver_status_interval</varname> milliseconds. Make sure
> + to set <varname>wal_receiver_status_interval</varname> less than the
> + <varname>wal_sender_timeout</varname> on the publisher, otherwise, the
> + <literal>walsender</literal> will repeatedly terminate due to timeout
> + error. If <varname>wal_receiver_status_interval</varname> is set to
> + zero, the apply worker doesn't send any feedback messages during the
> + <literal>min_apply_delay</literal> interval.
>
> I removed the parenthesis explanation about time-delayed logical replication.
> If you are reading the documentation and does not know what it means you should
> (a) read the logical replication chapter or (b) check the glossary (maybe a new
> entry should be added). I also removed the Standby status Update message but it
> is a low level detail; let's refer to it as feedback message as the other
> sentences do. I changed "literal" to "varname" that's the correct tag for
> parameters. I replace "period" with "interval" that was the previous
> terminology. IMO we should be uniform, use one or the other.
Adopted.
Also, I added the glossary for time-delayed replication (one for
applicable to both physical replication and logical replication).
Plus, I united the term "interval" to period, because it would clarify the type for this feature.
I think this is better.
> - The subscriber replication can be instructed to lag behind the publisher
> - side changes by specifying the <literal>min_apply_delay</literal>
> - subscription parameter. See <xref linkend="sql-createsubscription"/> for
> - details.
> + A logical replication subscription can delay the application of changes by
> + specifying the <literal>min_apply_delay</literal> subscription parameter.
> + See <xref linkend="sql-createsubscription"/> for details.
>
> This feature refers to a specific subscription, hence, "logical replication
> subscription" instead of "subscriber replication".
Adopted.
> + if (IsSet(opts->specified_opts, SUBOPT_MIN_APPLY_DELAY))
> + errorConflictingDefElem(defel, pstate);
> +
>
> Peter S referred to this missing piece of code too.
Added.
> -int
> +static int
> defGetMinApplyDelay(DefElem *def)
> {
>
> It seems you forgot static keyword.
Fixed.
> - elog(DEBUG2, "time-delayed replication for txid %u, min_apply_delay = %lld ms, Remaining wait time: %ld ms",
> - xid, (long long) MySubscription->minapplydelay, diffms);
> + elog(DEBUG2, "time-delayed replication for txid %u, min_apply_delay = " INT64_FORMAT " ms, remaining wait time: %ld ms",
> + xid, MySubscription->minapplydelay, diffms);
> int64 should use format modifier INT64_FORMAT.
Fixed.
> - (long) wal_receiver_status_interval * 1000,
> + wal_receiver_status_interval * 1000L,
>
> Cast is not required. I added a suffix to the constant.
Fixed.
> - elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write %X/%X, flush %X/%X in-delayed: %d",
> + elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write %X/%X, flush %X/%X, apply delay: %s",
> force,
> LSN_FORMAT_ARGS(recvpos),
> LSN_FORMAT_ARGS(writepos),
> LSN_FORMAT_ARGS(flushpos),
> - in_delayed_apply);
> + in_delayed_apply? "yes" : "no");
>
> It is better to use a string to represent the yes/no option.
Fixed.
> - gettext_noop("Min apply delay (ms)"));
> + gettext_noop("Min apply delay"));
>
> I don't know if it was discussed but we don't add units to headers. When I
> think about this parameter representation (internal and external), I decided to
> use the previous code because it provides a unit for external representation. I
> understand that using the same representation as recovery_min_apply_delay is
> good but the current code does not handle the external representation
> accordingly. (recovery_min_apply_delay uses the GUC machinery to adds the unit
> but for min_apply_delay, it doesn't).
Adopted.
> # Setup for streaming case
> -$node_publisher->append_conf('postgres.conf',
> +$node_publisher->append_conf('postgresql.conf',
> 'logical_decoding_mode = immediate');
> $node_publisher->reload;
>
> Fix configuration file name.
Fixed.
> Maybe tests should do a better job. I think check_apply_delay_time is fragile
> because it does not guarantee that time is not shifted. Time-delayed
> replication is a subscriber feature and to check its correctness it should
> check the logs.
>
> # Note that we cannot call check_apply_delay_log() here because there is a
> # possibility that the delay is skipped. The event happens when the WAL
> # replication between publisher and subscriber is delayed due to a mechanical
> # problem. The log output will be checked later - substantial delay-time case.
>
> If you might not use the logs for it, it should adjust the min_apply_delay, no?
Yes. Adjusted.
> It does not exercise the min_apply_delay vs parallel streaming mode.
>
> + /*
> + * The combination of parallel streaming mode and
> + * min_apply_delay is not allowed.
> + */
> + if (opts.streaming == LOGICALREP_STREAM_PARALLEL)
> + if ((IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) && opts.min_apply_delay > 0) ||
> + (!IsSet(opts.specified_opts, SUBOPT_MIN_APPLY_DELAY) && sub->minapplydelay > 0))
> + ereport(ERROR,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("cannot enable %s mode for subscription with %s",
> + "streaming = parallel", "min_apply_delay"));
> +
>
> Is this code correct? I also didn't like this message. "cannot enable streaming
> = parallel mode for subscription with min_apply_delay" is far from a good error
> message. How about refer parallelism to "parallel streaming mode".
Yes. opts is the input for alter command and sub object is the existing definition.
We need to check those combinations like when streaming is set to parallel
and min_apply_delay also gets set, then, min_apply_delay should not be bigger than 0, for example.
Besides, adopted your suggestion to improve the comments.
Attach the patch in [1]. Kindly have a look at it.
[1] - https://www.postgresql.org/message-id/TYCPR01MB8373DC1881F382B4703F26E0EDC99%40TYCPR01MB8373.jpnprd0...
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-02-06 11:56 Amit Kapila <[email protected]>
parent: Euler Taveira <[email protected]>
2 siblings, 1 reply; 40+ messages in thread
From: Amit Kapila @ 2023-02-06 11:56 UTC (permalink / raw)
To: Euler Taveira <[email protected]>; +Cc: [email protected] <[email protected]>; Peter Smith <[email protected]>; vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Dilip Kumar <[email protected]>; Melih Mutlu <[email protected]>; Andres Freund <[email protected]>; Marcos Pegoraro <[email protected]>; pgsql-hackers
On Tue, Jan 24, 2023 at 5:02 AM Euler Taveira <[email protected]> wrote:
>
>
> - elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write %X/%X, flush %X/%X in-delayed: %d",
> + elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write %X/%X, flush %X/%X, apply delay: %s",
> force,
> LSN_FORMAT_ARGS(recvpos),
> LSN_FORMAT_ARGS(writepos),
> LSN_FORMAT_ARGS(flushpos),
> - in_delayed_apply);
> + in_delayed_apply? "yes" : "no");
>
> It is better to use a string to represent the yes/no option.
>
I think it is better to be consistent with the existing force
parameter which is also boolean, otherwise, it will look odd.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Time delayed LR (WAS Re: logical replication restrictions)
@ 2023-02-06 13:21 Takamichi Osumi (Fujitsu) <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Takamichi Osumi (Fujitsu) @ 2023-02-06 13:21 UTC (permalink / raw)
To: 'Amit Kapila' <[email protected]>; Euler Taveira <[email protected]>; +Cc: Peter Smith <[email protected]>; vignesh C <[email protected]>; Kyotaro Horiguchi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>; Dilip Kumar <[email protected]>; Melih Mutlu <[email protected]>; Andres Freund <[email protected]>; Marcos Pegoraro <[email protected]>; pgsql-hackers
Hi,
On Monday, February 6, 2023 8:57 PM Amit Kapila <[email protected]> wrote:
> On Tue, Jan 24, 2023 at 5:02 AM Euler Taveira <[email protected]> wrote:
> >
> >
> > - elog(DEBUG2, "sending feedback (force %d) to recv %X/%X,
> write %X/%X, flush %X/%X in-delayed: %d",
> > + elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write
> > + %X/%X, flush %X/%X, apply delay: %s",
> > force,
> > LSN_FORMAT_ARGS(recvpos),
> > LSN_FORMAT_ARGS(writepos),
> > LSN_FORMAT_ARGS(flushpos),
> > - in_delayed_apply);
> > + in_delayed_apply? "yes" : "no");
> >
> > It is better to use a string to represent the yes/no option.
> >
>
> I think it is better to be consistent with the existing force parameter which is
> also boolean, otherwise, it will look odd.
Agreed. The latest patch v29 posted in [1] followed this suggestion.
Kindly have a look at it.
[1] - https://www.postgresql.org/message-id/TYCPR01MB8373A59E7B74AA4F96B62BEAEDDA9%40TYCPR01MB8373.jpnprd0...
Best Regards,
Takamichi Osumi
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v37 06/11] Add Incremental View Maintenance support
@ 2026-05-29 09:06 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Yugo Nagata @ 2026-05-29 09:06 UTC (permalink / raw)
In this implementation, AFTER triggers are used to collect
tuplestores containing transition table contents. When multiple tables
are changed, multiple AFTER triggers are invoked, then the final AFTER
trigger performs actual update of the matview. In addition, BEFORE
triggers are also used to handle global information for view
maintenance.
To calculate view deltas, we need both pre-state and post-state of base
tables. Post-update states are available in AFTER trigger, and pre-update
states can be calculated by removing inserted tuples and appending deleted
tuples. Insterted tuples are filtered using the snapshot taken before
table modification, and deleted tuples are contained in the old transition
table.
Incrementally Maintainable Materialized Views (IMMV) can contain
duplicated tuples.
This patch also allows self-join, simultaneous updates of more than
one base table, and multiple updates of the same base table.
---
src/backend/access/transam/xact.c | 8 +
src/backend/commands/createas.c | 709 +++++++
src/backend/commands/matview.c | 1835 ++++++++++++++++-
src/backend/commands/tablecmds.c | 4 +
.../utils/activity/wait_event_names.txt | 1 +
src/include/catalog/pg_proc.dat | 10 +
src/include/commands/createas.h | 4 +
src/include/commands/matview.h | 11 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/subsystemlist.h | 3 +
10 files changed, 2548 insertions(+), 38 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 5586fbe5b07..302a4822d50 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -37,6 +37,7 @@
#include "catalog/pg_enum.h"
#include "catalog/storage.h"
#include "commands/async.h"
+#include "commands/matview.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "common/pg_prng.h"
@@ -2322,6 +2323,9 @@ CommitTransaction(void)
CallXactCallbacks(is_parallel_worker ? XACT_EVENT_PARALLEL_PRE_COMMIT
: XACT_EVENT_PRE_COMMIT);
+ /* Store the transaction ID that updated the view incrementally */
+ AtPreCommit_IVM();
+
/*
* If this xact has started any unfinished parallel operation, clean up
* its workers, warning about leaked resources. (But we don't actually
@@ -2971,6 +2975,7 @@ AbortTransaction(void)
AtAbort_Notify();
AtEOXact_RelationMap(false, is_parallel_worker);
AtAbort_Twophase();
+ AtAbort_IVM(InvalidSubTransactionId);
/*
* Advertise the fact that we aborted in pg_xact (assuming that we got as
@@ -5301,6 +5306,9 @@ AbortSubTransaction(void)
UnlockBuffers();
+ /* Clean up hash entries for incremental view maintenance */
+ AtAbort_IVM(s->subTransactionId);
+
/* Reset WAL record construction state */
XLogResetInsertion();
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 6dbb831ca89..a499688b79d 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -29,18 +29,32 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/namespace.h"
+#include "catalog/index.h"
+#include "catalog/pg_am_d.h"
+#include "catalog/pg_constraint.h"
+#include "catalog/pg_inherits.h"
+#include "catalog/pg_trigger.h"
#include "catalog/toasting.h"
#include "commands/createas.h"
+#include "commands/defrem.h"
#include "commands/matview.h"
#include "commands/prepare.h"
#include "commands/tablecmds.h"
+#include "commands/tablespace.h"
+#include "commands/trigger.h"
#include "commands/view.h"
#include "executor/execdesc.h"
#include "executor/executor.h"
+#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
#include "nodes/queryjumble.h"
+#include "optimizer/optimizer.h"
+#include "optimizer/prep.h"
#include "parser/analyze.h"
+#include "parser/parser.h"
+#include "parser/parsetree.h"
+#include "parser/parse_clause.h"
#include "rewrite/rewriteHandler.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
@@ -70,6 +84,12 @@ static bool intorel_receive(TupleTableSlot *slot, DestReceiver *self);
static void intorel_shutdown(DestReceiver *self);
static void intorel_destroy(DestReceiver *self);
+static void CreateIvmTriggersOnBaseTablesRecurse(Query *qry, Node *node, Oid matviewOid,
+ List **relids, bool ex_lock);
+static void CreateIvmTrigger(Oid relOid, Oid viewOid, int16 type, int16 timing, bool ex_lock);
+static void check_ivm_restriction(Node *node);
+static bool check_ivm_restriction_walker(Node *node, void *context);
+static Bitmapset *get_primary_key_attnos_from_query(Query *query, List **constraintList);
/*
* create_ctas_internal
@@ -277,6 +297,18 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
into->skipData = true;
}
+ if (is_matview && into->ivm)
+ {
+ /* check if the query is supported in IMMV definition */
+ if (contain_mutable_functions((Node *) query))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("mutable function is not supported on incrementally maintainable materialized view"),
+ errhint("functions must be marked IMMUTABLE")));
+
+ check_ivm_restriction((Node *) query);
+ }
+
if (into->skipData)
{
/*
@@ -287,15 +319,36 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
*/
address = create_ctas_nodata(query->targetList, into);
+ /*
+ * For IVM, mark the matview before refresh so RelationIsIVM()
+ * returns true inside RefreshMatViewByOid.
+ */
+ if (is_matview && into->ivm)
+ {
+ Relation matviewRel = table_open(address.objectId, NoLock);
+
+ SetMatViewIVMState(matviewRel, true);
+ table_close(matviewRel, NoLock);
+ CommandCounterIncrement();
+ }
+
/*
* For materialized views, reuse the REFRESH logic, which locks down
* security-restricted operations and restricts the search_path. This
* reduces the chance that a subsequent refresh will fail.
*/
if (do_refresh)
+ {
RefreshMatViewByOid(address.objectId, true, false, false,
pstate->p_sourcetext, qc);
+ if (is_matview && into->ivm && IsolationUsesXactSnapshot())
+ ereport(WARNING,
+ (errmsg("inconsistent view can be created in isolation level SERIALIZABLE or REPEATABLE READ"),
+ errdetail("The view may not include effects of a concurrent transaction."),
+ errhint("The view with incremental maintenance should be created in isolation level READ COMMITTED, "
+ "or refreshed manually to make sure the view is consistent.")));
+ }
}
else
{
@@ -635,3 +688,659 @@ intorel_destroy(DestReceiver *self)
{
pfree(self);
}
+
+/*
+ * CreateIvmTriggersOnBaseTables -- create IVM triggers on all base tables
+ */
+void
+CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid)
+{
+ List *relids = NIL;
+ bool ex_lock = false;
+ RangeTblEntry *rte;
+
+ /* Immediately return if we don't have any base tables. */
+ if (list_length(qry->rtable) < 1)
+ return;
+
+ /*
+ * If the view has more than one base tables, we need an exclusive lock
+ * on the view so that the view would be maintained serially to avoid
+ * the inconsistency that occurs when two base tables are modified in
+ * concurrent transactions. However, if the view has only one table,
+ * we can use a weaker lock.
+ *
+ * The type of lock should be determined here, because if we check the
+ * view definition at maintenance time, we need to acquire a weaker lock,
+ * and upgrading the lock level after this increases probability of
+ * deadlock.
+ */
+
+ rte = list_nth(qry->rtable, 0);
+ if (list_length(qry->rtable) > 1 || rte->rtekind != RTE_RELATION)
+ ex_lock = true;
+
+ CreateIvmTriggersOnBaseTablesRecurse(qry, (Node *)qry, matviewOid, &relids, ex_lock);
+
+ list_free(relids);
+}
+
+static void
+CreateIvmTriggersOnBaseTablesRecurse(Query *qry, Node *node, Oid matviewOid,
+ List **relids, bool ex_lock)
+{
+ if (node == NULL)
+ return;
+
+ /* This can recurse, so check for excessive recursion */
+ check_stack_depth();
+
+ switch (nodeTag(node))
+ {
+ case T_Query:
+ {
+ Query *query = (Query *) node;
+
+ CreateIvmTriggersOnBaseTablesRecurse(qry, (Node *)query->jointree, matviewOid, relids, ex_lock);
+ }
+ break;
+
+ case T_RangeTblRef:
+ {
+ int rti = ((RangeTblRef *) node)->rtindex;
+ RangeTblEntry *rte = rt_fetch(rti, qry->rtable);
+
+ if (rte->rtekind == RTE_RELATION && !list_member_oid(*relids, rte->relid))
+ {
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_INSERT, TRIGGER_TYPE_BEFORE, ex_lock);
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_DELETE, TRIGGER_TYPE_BEFORE, ex_lock);
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_UPDATE, TRIGGER_TYPE_BEFORE, ex_lock);
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_TRUNCATE, TRIGGER_TYPE_BEFORE, true);
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_INSERT, TRIGGER_TYPE_AFTER, ex_lock);
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_DELETE, TRIGGER_TYPE_AFTER, ex_lock);
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_UPDATE, TRIGGER_TYPE_AFTER, ex_lock);
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_TRUNCATE, TRIGGER_TYPE_AFTER, true);
+
+ *relids = lappend_oid(*relids, rte->relid);
+ }
+ }
+ break;
+
+ case T_FromExpr:
+ {
+ FromExpr *f = (FromExpr *) node;
+ ListCell *l;
+
+ foreach(l, f->fromlist)
+ CreateIvmTriggersOnBaseTablesRecurse(qry, lfirst(l), matviewOid, relids, ex_lock);
+ }
+ break;
+
+ case T_JoinExpr:
+ {
+ JoinExpr *j = (JoinExpr *) node;
+
+ CreateIvmTriggersOnBaseTablesRecurse(qry, j->larg, matviewOid, relids, ex_lock);
+ CreateIvmTriggersOnBaseTablesRecurse(qry, j->rarg, matviewOid, relids, ex_lock);
+ }
+ break;
+
+ default:
+ elog(ERROR, "unrecognized node type: %d", (int) nodeTag(node));
+ }
+}
+
+/*
+ * CreateIvmTrigger -- create IVM trigger on a base table
+ */
+static void
+CreateIvmTrigger(Oid relOid, Oid viewOid, int16 type, int16 timing, bool ex_lock)
+{
+ ObjectAddress refaddr;
+ ObjectAddress address;
+ CreateTrigStmt *ivm_trigger;
+ List *transitionRels = NIL;
+
+ Assert(timing == TRIGGER_TYPE_BEFORE || timing == TRIGGER_TYPE_AFTER);
+
+ refaddr.classId = RelationRelationId;
+ refaddr.objectId = viewOid;
+ refaddr.objectSubId = 0;
+
+ ivm_trigger = makeNode(CreateTrigStmt);
+ ivm_trigger->relation = NULL;
+ ivm_trigger->row = false;
+
+ ivm_trigger->timing = timing;
+ ivm_trigger->events = type;
+
+ switch (type)
+ {
+ case TRIGGER_TYPE_INSERT:
+ ivm_trigger->trigname = (timing == TRIGGER_TYPE_BEFORE ? "IVM_trigger_ins_before" : "IVM_trigger_ins_after");
+ break;
+ case TRIGGER_TYPE_DELETE:
+ ivm_trigger->trigname = (timing == TRIGGER_TYPE_BEFORE ? "IVM_trigger_del_before" : "IVM_trigger_del_after");
+ break;
+ case TRIGGER_TYPE_UPDATE:
+ ivm_trigger->trigname = (timing == TRIGGER_TYPE_BEFORE ? "IVM_trigger_upd_before" : "IVM_trigger_upd_after");
+ break;
+ case TRIGGER_TYPE_TRUNCATE:
+ ivm_trigger->trigname = (timing == TRIGGER_TYPE_BEFORE ? "IVM_trigger_truncate_before" : "IVM_trigger_truncate_after");
+ break;
+ default:
+ elog(ERROR, "unsupported trigger type");
+ }
+
+ if (timing == TRIGGER_TYPE_AFTER)
+ {
+ if (type == TRIGGER_TYPE_INSERT || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "__ivm_newtable";
+ n->isNew = true;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+ if (type == TRIGGER_TYPE_DELETE || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "__ivm_oldtable";
+ n->isNew = false;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+ }
+
+ /*
+ * XXX: When using DELETE or UPDATE, we must use exclusive lock for now
+ * because apply_old_delta(_with_count) uses ctid to identify the tuple
+ * to be deleted/deleted, but doesn't work in concurrent situations.
+ *
+ * If the view doesn't have aggregate, distinct, or tuple duplicate,
+ * then it would work even in concurrent situations. However, we don't have
+ * any way to guarantee the view has a unique key before opening the IMMV
+ * at the maintenance time because users may drop the unique index.
+ */
+
+ if (type == TRIGGER_TYPE_DELETE || type == TRIGGER_TYPE_UPDATE)
+ ex_lock = true;
+
+ ivm_trigger->funcname =
+ (timing == TRIGGER_TYPE_BEFORE ? SystemFuncName("IVM_immediate_before") : SystemFuncName("IVM_immediate_maintenance"));
+
+ ivm_trigger->columns = NIL;
+ ivm_trigger->transitionRels = transitionRels;
+ ivm_trigger->whenClause = NULL;
+ ivm_trigger->isconstraint = false;
+ ivm_trigger->deferrable = false;
+ ivm_trigger->initdeferred = false;
+ ivm_trigger->constrrel = NULL;
+ ivm_trigger->args = list_make2(
+ makeString(DatumGetPointer(DirectFunctionCall1(oidout, ObjectIdGetDatum(viewOid)))),
+ makeString(DatumGetPointer(DirectFunctionCall1(boolout, BoolGetDatum(ex_lock))))
+ );
+
+ address = CreateTrigger(ivm_trigger, NULL, relOid, InvalidOid, InvalidOid,
+ InvalidOid, InvalidOid, InvalidOid, NULL, true, false);
+
+ recordDependencyOn(&address, &refaddr, DEPENDENCY_AUTO);
+
+ /* Make changes-so-far visible */
+ CommandCounterIncrement();
+}
+
+/*
+ * check_ivm_restriction --- look for specify nodes in the query tree
+ */
+static void
+check_ivm_restriction(Node *node)
+{
+ check_ivm_restriction_walker(node, NULL);
+}
+
+static bool
+check_ivm_restriction_walker(Node *node, void *context)
+{
+ if (node == NULL)
+ return false;
+
+ /*
+ * We currently don't support Sub-Query.
+ */
+ if (IsA(node, SubPlan) || IsA(node, SubLink))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("subquery is not supported on incrementally maintainable materialized view")));
+
+ /* This can recurse, so check for excessive recursion */
+ check_stack_depth();
+
+ switch (nodeTag(node))
+ {
+ case T_Query:
+ {
+ Query *qry = (Query *)node;
+ ListCell *lc;
+ List *vars;
+
+ /* if contained CTE, return error */
+ if (qry->cteList != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("CTE is not supported on incrementally maintainable materialized view")));
+ if (qry->havingQual != NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg(" HAVING clause is not supported on incrementally maintainable materialized view")));
+ if (qry->sortClause != NIL) /* There is a possibility that we don't need to return an error */
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("ORDER BY clause is not supported on incrementally maintainable materialized view")));
+ if (qry->limitOffset != NULL || qry->limitCount != NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
+ if (qry->distinctClause)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
+ if (qry->hasDistinctOn)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("DISTINCT ON is not supported on incrementally maintainable materialized view")));
+ if (qry->hasWindowFuncs)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("window functions are not supported on incrementally maintainable materialized view")));
+ if (qry->groupingSets != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("GROUPING SETS, ROLLUP, or CUBE clauses is not supported on incrementally maintainable materialized view")));
+ if (qry->setOperations != NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("UNION/INTERSECT/EXCEPT statements are not supported on incrementally maintainable materialized view")));
+ if (list_length(qry->targetList) == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("empty target list is not supported on incrementally maintainable materialized view")));
+ if (qry->rowMarks != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("FOR UPDATE/SHARE clause is not supported on incrementally maintainable materialized view")));
+
+ /* system column restrictions */
+ vars = pull_vars_of_level((Node *) qry, 0);
+ foreach(lc, vars)
+ {
+ if (IsA(lfirst(lc), Var))
+ {
+ Var *var = (Var *) lfirst(lc);
+ /* if system column, return error */
+ if (var->varattno < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("system column is not supported on incrementally maintainable materialized view")));
+ }
+ }
+
+ /* check if type in the target list had an equality operator */
+ foreach(lc, qry->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Oid atttype = exprType((Node *) tle->expr);
+ Oid opclass;
+
+ opclass = GetDefaultOpClass(atttype, BTREE_AM_OID);
+ if (!OidIsValid(opclass))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("data type %s has no default operator class for access method \"%s\"",
+ format_type_be(atttype), "btree")));
+ }
+
+ /* restrictions for rtable */
+ foreach(lc, qry->rtable)
+ {
+ RangeTblEntry *rte = (RangeTblEntry *) lfirst(lc);
+
+ if (rte->subquery)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("subquery is not supported on incrementally maintainable materialized view")));
+
+ if (rte->tablesample != NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("TABLESAMPLE clause is not supported on incrementally maintainable materialized view")));
+
+ if (rte->relkind == RELKIND_PARTITIONED_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("partitioned table is not supported on incrementally maintainable materialized view")));
+
+ if (rte->relkind == RELKIND_RELATION && has_superclass(rte->relid))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("partitions is not supported on incrementally maintainable materialized view")));
+
+ if (rte->relkind == RELKIND_RELATION && find_inheritance_children(rte->relid, NoLock) != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("inheritance parent is not supported on incrementally maintainable materialized view")));
+
+ if (rte->relkind == RELKIND_FOREIGN_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("foreign table is not supported on incrementally maintainable materialized view")));
+
+ if (rte->relkind == RELKIND_VIEW ||
+ rte->relkind == RELKIND_MATVIEW)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("VIEW or MATERIALIZED VIEW is not supported on incrementally maintainable materialized view")));
+
+ if (rte->rtekind == RTE_VALUES)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("VALUES is not supported on incrementally maintainable materialized view")));
+
+ }
+
+ query_tree_walker(qry, check_ivm_restriction_walker, NULL, QTW_IGNORE_RANGE_TABLE);
+
+ break;
+ }
+ case T_TargetEntry:
+ {
+ TargetEntry *tle = (TargetEntry *)node;
+ if (isIvmName(tle->resname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column name %s is not supported on incrementally maintainable materialized view", tle->resname)));
+
+ expression_tree_walker(node, check_ivm_restriction_walker, NULL);
+ break;
+ }
+ case T_JoinExpr:
+ {
+ JoinExpr *joinexpr = (JoinExpr *)node;
+
+ if (joinexpr->jointype > JOIN_INNER)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("OUTER JOIN is not supported on incrementally maintainable materialized view")));
+
+ expression_tree_walker(node, check_ivm_restriction_walker, NULL);
+ }
+ break;
+ case T_Aggref:
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("aggregate function is not supported on incrementally maintainable materialized view")));
+ break;
+ default:
+ expression_tree_walker(node, check_ivm_restriction_walker, (void *) context);
+ break;
+ }
+ return false;
+}
+
+/*
+ * CreateIndexOnIMMV
+ *
+ * Create a unique index on incremental maintainable materialized view.
+ * If the view definition query has a GROUP BY clause, the index is created
+ * on the columns of GROUP BY expressions. Otherwise, if the view contains
+ * all primary key attributes of its base tables in the target list, the index
+ * is created on these attributes. In other cases, no index is created.
+ */
+void
+CreateIndexOnIMMV(Query *query, Relation matviewRel)
+{
+ ListCell *lc;
+ IndexStmt *index;
+ ObjectAddress address;
+ List *constraintList = NIL;
+ char idxname[NAMEDATALEN];
+ List *indexoidlist = RelationGetIndexList(matviewRel);
+ ListCell *indexoidscan;
+ Bitmapset *key_attnos;
+
+ snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
+
+ index = makeNode(IndexStmt);
+
+ index->unique = true;
+ index->primary = false;
+ index->isconstraint = false;
+ index->deferrable = false;
+ index->initdeferred = false;
+ index->idxname = idxname;
+ index->relation =
+ makeRangeVar(get_namespace_name(RelationGetNamespace(matviewRel)),
+ pstrdup(RelationGetRelationName(matviewRel)),
+ -1);
+ index->accessMethod = DEFAULT_INDEX_TYPE;
+ index->options = NIL;
+ index->tableSpace = get_tablespace_name(matviewRel->rd_rel->reltablespace);
+ index->whereClause = NULL;
+ index->indexParams = NIL;
+ index->indexIncludingParams = NIL;
+ index->excludeOpNames = NIL;
+ index->idxcomment = NULL;
+ index->indexOid = InvalidOid;
+ index->oldNumber = InvalidRelFileNumber;
+ index->oldCreateSubid = InvalidSubTransactionId;
+ index->oldFirstRelfilelocatorSubid = InvalidSubTransactionId;
+ index->transformed = true;
+ index->concurrent = false;
+ index->if_not_exists = false;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns. "),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
+
+ /* If we have a compatible index, we don't need to create another. */
+ foreach(indexoidscan, indexoidlist)
+ {
+ Oid indexoid = lfirst_oid(indexoidscan);
+ Relation indexRel;
+ bool hasCompatibleIndex = false;
+
+ indexRel = index_open(indexoid, AccessShareLock);
+
+ if (CheckIndexCompatible(indexRel->rd_id,
+ index->accessMethod,
+ index->indexParams,
+ index->excludeOpNames,
+ false))
+ hasCompatibleIndex = true;
+
+ index_close(indexRel, AccessShareLock);
+
+ if (hasCompatibleIndex)
+ return;
+ }
+
+ address = DefineIndex(NULL,
+ RelationGetRelid(matviewRel),
+ index,
+ InvalidOid,
+ InvalidOid,
+ InvalidOid,
+ -1,
+ false, true, false, false, true);
+
+ ereport(NOTICE,
+ (errmsg("created index \"%s\" on materialized view \"%s\"",
+ idxname, RelationGetRelationName(matviewRel))));
+
+ /*
+ * Make dependencies so that the index is dropped if any base tables'
+ * primary key is dropped.
+ */
+ foreach(lc, constraintList)
+ {
+ Oid constraintOid = lfirst_oid(lc);
+ ObjectAddress refaddr;
+
+ refaddr.classId = ConstraintRelationId;
+ refaddr.objectId = constraintOid;
+ refaddr.objectSubId = 0;
+
+ recordDependencyOn(&address, &refaddr, DEPENDENCY_NORMAL);
+ }
+}
+
+
+/*
+ * get_primary_key_attnos_from_query
+ *
+ * Identify the columns in base tables' primary keys in the target list.
+ *
+ * Returns a Bitmapset of the column attnos of the primary key's columns of
+ * tables that used in the query. The attnos are offset by
+ * FirstLowInvalidHeapAttributeNumber as same as get_primary_key_attnos.
+ *
+ * If any table has no primary key or any primary key's columns is not in
+ * the target list, return NULL. We also return NULL if any pkey constraint
+ * is deferrable.
+ *
+ * constraintList is set to a list of the OIDs of the pkey constraints.
+ */
+static Bitmapset *
+get_primary_key_attnos_from_query(Query *query, List **constraintList)
+{
+ List *key_attnos_list = NIL;
+ ListCell *lc;
+ int i;
+ Bitmapset *keys = NULL;
+ Relids rels_in_from;
+
+ /*
+ * Collect primary key attributes from all tables used in query. The key attributes
+ * sets for each table are stored in key_attnos_list in order by RTE index.
+ */
+ foreach(lc, query->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc);
+ Bitmapset *key_attnos;
+ bool has_no_pkey = false;
+
+ /* for tables, call get_primary_key_attnos */
+ if (r->rtekind == RTE_RELATION)
+ {
+ Oid constraintOid;
+ key_attnos = get_primary_key_attnos(r->relid, false, &constraintOid);
+ *constraintList = lappend_oid(*constraintList, constraintOid);
+ has_no_pkey = (key_attnos == NULL);
+ }
+ /*
+ * Ignore join rels, because they are flatten later by
+ * flatten_join_alias_vars(). Store NULL into key_attnos_list
+ * as a dummy.
+ */
+ else if (r->rtekind == RTE_JOIN)
+ {
+ key_attnos = NULL;
+ }
+ /* for other RTEs, store NULL into key_attnos_list */
+ else
+ has_no_pkey = true;
+
+ /*
+ * If any table or subquery has no primary key or its pkey constraint is deferrable,
+ * we cannot get key attributes for this query, so return NULL.
+ */
+ if (has_no_pkey)
+ return NULL;
+
+ key_attnos_list = lappend(key_attnos_list, key_attnos);
+ }
+
+ /* Collect key attributes appearing in the target list */
+ i = 1;
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) flatten_join_alias_vars(NULL, query, lfirst(lc));
+
+ if (IsA(tle->expr, Var))
+ {
+ Var *var = (Var*) tle->expr;
+ Bitmapset *key_attnos = list_nth(key_attnos_list, var->varno - 1);
+
+ /* check if this attribute is from a base table's primary key */
+ if (bms_is_member(var->varattno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ /*
+ * Remove found key attributes from key_attnos_list, and add this
+ * to the result list.
+ */
+ key_attnos = bms_del_member(key_attnos, var->varattno - FirstLowInvalidHeapAttributeNumber);
+ if (bms_is_empty(key_attnos))
+ {
+ key_attnos_list = list_delete_nth_cell(key_attnos_list, var->varno - 1);
+ key_attnos_list = list_insert_nth(key_attnos_list, var->varno - 1, NULL);
+ }
+ keys = bms_add_member(keys, i - FirstLowInvalidHeapAttributeNumber);
+ }
+ }
+ i++;
+ }
+
+ /* Collect RTE indexes of relations appearing in the FROM clause */
+ rels_in_from = get_relids_in_jointree((Node *) query->jointree, false, false);
+
+ /*
+ * Check if all key attributes of relations in FROM are appearing in the target
+ * list. If an attribute remains in key_attnos_list in spite of the table is used
+ * in FROM clause, the target is missing this key attribute, so we return NULL.
+ */
+ i = 1;
+ foreach(lc, key_attnos_list)
+ {
+ Bitmapset *bms = (Bitmapset *)lfirst(lc);
+ if (!bms_is_empty(bms) && bms_is_member(i, rels_in_from))
+ return NULL;
+ i++;
+ }
+
+ return keys;
+}
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index f7d8007f796..eaf39f80cd3 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -23,24 +23,36 @@
#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/pg_am.h"
+#include "catalog/pg_depend.h"
+#include "catalog/pg_trigger.h"
#include "catalog/pg_opclass.h"
#include "commands/matview.h"
#include "commands/repack.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
+#include "commands/createas.h"
#include "executor/executor.h"
#include "executor/spi.h"
+#include "executor/tstoreReceiver.h"
#include "miscadmin.h"
+#include "nodes/makefuncs.h"
+#include "parser/analyze.h"
+#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
+#include "parser/parse_relation.h"
#include "pgstat.h"
#include "rewrite/rewriteHandler.h"
+#include "rewrite/rowsecurity.h"
#include "storage/lmgr.h"
+#include "storage/lwlock.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
-
+#include "utils/typcache.h"
typedef struct
{
@@ -53,6 +65,97 @@ typedef struct
BulkInsertState bistate; /* bulk insert state */
} DR_transientrel;
+#define MV_INIT_QUERYHASHSIZE 16
+
+/*
+ * MV_TriggerHashEntry
+ *
+ * Hash entry for base tables on which IVM trigger is invoked
+ */
+typedef struct MV_TriggerHashEntry
+{
+ Oid matview_id; /* OID of the materialized view */
+ int before_trig_count; /* count of before triggers invoked */
+ int after_trig_count; /* count of after triggers invoked */
+
+ Snapshot snapshot; /* Snapshot just before table change */
+
+ List *tables; /* List of MV_TriggerTable */
+ bool has_old; /* tuples are deleted from any table? */
+ bool has_new; /* tuples are inserted into any table? */
+
+ /*
+ * List of sub-transaction IDs that incrementally updated the view.
+ * This list is maintained through a transaction, and an ID is removed
+ * when a sub-transaction is aborted. If any ID is left when the
+ * transaction is committed, this means the view is incrementally
+ * updated in this transaction.
+ */
+ List *subxids;
+} MV_TriggerHashEntry;
+
+/*
+ * MV_TriggerTable
+ *
+ * IVM related data for tables on which the trigger is invoked.
+ */
+typedef struct MV_TriggerTable
+{
+ Oid table_id; /* OID of the modified table */
+ List *old_tuplestores; /* tuplestores for deleted tuples */
+ List *new_tuplestores; /* tuplestores for inserted tuples */
+
+ List *rte_indexes; /* List of RTE index of the modified table */
+ RangeTblEntry *original_rte; /* the original RTE saved before rewriting query */
+
+ Relation rel; /* relation of the modified table */
+ TupleTableSlot *slot; /* for checking visibility in the pre-state table */
+} MV_TriggerTable;
+
+typedef struct DroppedImmvInfo
+{
+ Oid immv_oid;
+ SubTransactionId subxid;
+} DroppedImmvInfo;
+
+static HTAB *mv_trigger_info = NULL;
+static HTAB *dropped_immv = NULL;
+
+static HTAB *LastIvmUpdateHash;
+
+#define LAST_IVM_UPDATE_HASH_SIZE 256
+
+static void IvmShmemRequest(void *arg);
+
+/* hash table entries */
+typedef struct LastIvmUpdateEntry
+{
+ Oid oid; /* hash key */
+ FullTransactionId last_ivm_update;
+} LastIvmUpdateEntry;
+
+const ShmemCallbacks IvmShmemCallbacks = {
+ .request_fn = IvmShmemRequest,
+};
+
+static void
+IvmShmemRequest(void *arg)
+{
+ ShmemRequestHash(.name = "LastIvmUpdate hash",
+ .nelems = LAST_IVM_UPDATE_HASH_SIZE,
+ .ptr = &LastIvmUpdateHash,
+ .hash_info.keysize = sizeof(Oid),
+ .hash_info.entrysize = sizeof(LastIvmUpdateEntry),
+ .hash_flags = HASH_ELEM | HASH_BLOBS,
+ );
+}
+
+static bool in_delta_calculation = false;
+
+/* ENR name for materialized view delta */
+#define NEW_DELTA_ENRNAME "new_delta"
+#define OLD_DELTA_ENRNAME "old_delta"
+
static int matview_maintenance_depth = 0;
static void transientrel_startup(DestReceiver *self, int operation, TupleDesc typeinfo);
@@ -60,6 +163,8 @@ static bool transientrel_receive(TupleTableSlot *slot, DestReceiver *self);
static void transientrel_shutdown(DestReceiver *self);
static void transientrel_destroy(DestReceiver *self);
static uint64 refresh_matview_datafill(DestReceiver *dest, Query *query,
+ QueryEnvironment *queryEnv,
+ TupleDesc *resultTupleDesc,
const char *queryString, bool is_create);
static void refresh_by_match_merge(Oid matviewOid, Oid tempOid, Oid relowner,
int save_sec_context);
@@ -67,6 +172,41 @@ static void refresh_by_heap_swap(Oid matviewOid, Oid OIDNewHeap, char relpersist
static bool is_usable_unique_index(Relation indexRel);
static void OpenMatViewIncrementalMaintenance(void);
static void CloseMatViewIncrementalMaintenance(void);
+static Query *get_matview_query(Relation matviewRel);
+
+static Query *rewrite_query_for_preupdate_state(Query *query, List *tables,
+ ParseState *pstate, Oid matviewid);
+static void register_delta_ENRs(ParseState *pstate, Query *query, List *tables);
+static char *make_subquery_targetlist_from_table(MV_TriggerTable *table);
+static char *make_delta_enr_name(const char *prefix, Oid relid, int count);
+static RangeTblEntry *get_prestate_rte(RangeTblEntry *rte, MV_TriggerTable *table,
+ QueryEnvironment *queryEnv, Oid matviewid);
+static RangeTblEntry *makeDeltaTable(RangeTblEntry *rte, MV_TriggerTable *table,
+ bool is_new, QueryEnvironment *queryEnv);
+static Query *rewrite_query_for_counting(Query *query, ParseState *pstate);
+
+static void calc_delta(MV_TriggerTable *table, int rte_index, Query *query,
+ DestReceiver *dest_old, DestReceiver *dest_new,
+ TupleDesc *tupdesc_old, TupleDesc *tupdesc_new,
+ QueryEnvironment *queryEnv);
+static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte_index);
+
+static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
+ TupleDesc tupdesc_old, TupleDesc tupdesc_new,
+ Query *query);
+static void apply_old_delta(const char *matviewname, const char *deltaname_old,
+ List *keys);
+static void apply_new_delta(const char *matviewname, const char *deltaname_new,
+ StringInfo target_list);
+static char *get_matching_condition_string(List *keys);
+static void generate_equal(StringInfo querybuf, Oid opttype,
+ const char *leftop, const char *rightop);
+
+static void mv_InitHashTables(void);
+static void clean_up_IVM_hash_entry(MV_TriggerHashEntry *entry, bool is_abort,
+ SubTransactionId subxid);
+static void setLastUpdateXid(Oid immv_oid, FullTransactionId xid);
+static FullTransactionId getLastUpdateXid(Oid immv_oid);
/*
* SetMatViewPopulatedState
@@ -108,6 +248,46 @@ SetMatViewPopulatedState(Relation relation, bool newstate)
CommandCounterIncrement();
}
+/*
+ * SetMatViewIVMState
+ * Mark a materialized view as IVM, or not.
+ *
+ * NOTE: caller must be holding an appropriate lock on the relation.
+ */
+void
+SetMatViewIVMState(Relation relation, bool newstate)
+{
+ Relation pgrel;
+ HeapTuple tuple;
+
+ Assert(relation->rd_rel->relkind == RELKIND_MATVIEW);
+
+ /*
+ * Update relation's pg_class entry. Crucial side-effect: other backends
+ * (and this one too!) are sent SI message to make them rebuild relcache
+ * entries.
+ */
+ pgrel = table_open(RelationRelationId, RowExclusiveLock);
+ tuple = SearchSysCacheCopy1(RELOID,
+ ObjectIdGetDatum(RelationGetRelid(relation)));
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for relation %u",
+ RelationGetRelid(relation));
+
+ ((Form_pg_class) GETSTRUCT(tuple))->relisivm = newstate;
+
+ CatalogTupleUpdate(pgrel, &tuple->t_self, tuple);
+
+ heap_freetuple(tuple);
+ table_close(pgrel, RowExclusiveLock);
+
+ /*
+ * Advance command counter to make the updated pg_class row locally
+ * visible.
+ */
+ CommandCounterIncrement();
+}
+
/*
* ExecRefreshMatView -- execute a REFRESH MATERIALIZED VIEW command
*
@@ -166,8 +346,6 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
QueryCompletion *qc)
{
Relation matviewRel;
- RewriteRule *rule;
- List *actions;
Query *dataQuery;
Oid tableSpace;
Oid relowner;
@@ -178,6 +356,7 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
int save_sec_context;
int save_nestlevel;
ObjectAddress address;
+ bool oldPopulated;
matviewRel = table_open(matviewOid, NoLock);
relowner = matviewRel->rd_rel->relowner;
@@ -193,6 +372,8 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
save_nestlevel = NewGUCNestLevel();
RestrictSearchPath();
+ oldPopulated = RelationIsPopulated(matviewRel);
+
/* Make sure it is a materialized view. */
if (matviewRel->rd_rel->relkind != RELKIND_MATVIEW)
ereport(ERROR,
@@ -213,32 +394,7 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
errmsg("%s options %s and %s cannot be used together",
"REFRESH", "CONCURRENTLY", "WITH NO DATA")));
- /*
- * Check that everything is correct for a refresh. Problems at this point
- * are internal errors, so elog is sufficient.
- */
- if (matviewRel->rd_rel->relhasrules == false ||
- matviewRel->rd_rules->numLocks < 1)
- elog(ERROR,
- "materialized view \"%s\" is missing rewrite information",
- RelationGetRelationName(matviewRel));
-
- if (matviewRel->rd_rules->numLocks > 1)
- elog(ERROR,
- "materialized view \"%s\" has too many rules",
- RelationGetRelationName(matviewRel));
-
- rule = matviewRel->rd_rules->rules[0];
- if (rule->event != CMD_SELECT || !(rule->isInstead))
- elog(ERROR,
- "the rule for materialized view \"%s\" is not a SELECT INSTEAD OF rule",
- RelationGetRelationName(matviewRel));
-
- actions = rule->actions;
- if (list_length(actions) != 1)
- elog(ERROR,
- "the rule for materialized view \"%s\" is not a single action",
- RelationGetRelationName(matviewRel));
+ dataQuery = get_matview_query(matviewRel);
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -275,12 +431,6 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
errhint("Create a unique index with no WHERE clause on one or more columns of the materialized view.")));
}
- /*
- * The stored query was rewritten at the time of the MV definition, but
- * has not been scribbled on by the planner.
- */
- dataQuery = linitial_node(Query, actions);
-
/*
* Check for active uses of the relation in the current transaction, such
* as open scans.
@@ -310,6 +460,90 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
relpersistence = matviewRel->rd_rel->relpersistence;
}
+ /* delete IMMV triggers. */
+ if (RelationIsIVM(matviewRel) && skipData)
+ {
+ Relation tgRel;
+ Relation depRel;
+ ScanKeyData key;
+ SysScanDesc scan;
+ HeapTuple tup;
+ ObjectAddresses *immv_triggers;
+
+ immv_triggers = new_object_addresses();
+
+ tgRel = table_open(TriggerRelationId, RowExclusiveLock);
+ depRel = table_open(DependRelationId, RowExclusiveLock);
+
+ /* search triggers that depends on IMMV. */
+ ScanKeyInit(&key,
+ Anum_pg_depend_refobjid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(matviewOid));
+ scan = systable_beginscan(depRel, DependReferenceIndexId, true,
+ NULL, 1, &key);
+ while ((tup = systable_getnext(scan)) != NULL)
+ {
+ ObjectAddress obj;
+ Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (foundDep->classid == TriggerRelationId)
+ {
+ HeapTuple tgtup;
+ ScanKeyData tgkey[1];
+ SysScanDesc tgscan;
+ Form_pg_trigger tgform;
+
+ /* Find the trigger name. */
+ ScanKeyInit(&tgkey[0],
+ Anum_pg_trigger_oid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(foundDep->objid));
+
+ tgscan = systable_beginscan(tgRel, TriggerOidIndexId, true,
+ NULL, 1, tgkey);
+ tgtup = systable_getnext(tgscan);
+ if (!HeapTupleIsValid(tgtup))
+ elog(ERROR, "could not find tuple for immv trigger %u", foundDep->objid);
+
+ tgform = (Form_pg_trigger) GETSTRUCT(tgtup);
+
+ /* If trigger is created by IMMV, delete it. */
+ if (strncmp(NameStr(tgform->tgname), "IVM_trigger_", 12) == 0)
+ {
+ obj.classId = foundDep->classid;
+ obj.objectId = foundDep->objid;
+ obj.objectSubId = foundDep->refobjsubid;
+ add_exact_object_address(&obj, immv_triggers);
+ }
+ systable_endscan(tgscan);
+ }
+ }
+ systable_endscan(scan);
+
+ performMultipleDeletions(immv_triggers, DROP_RESTRICT, PERFORM_DELETION_INTERNAL);
+
+ table_close(depRel, RowExclusiveLock);
+ table_close(tgRel, RowExclusiveLock);
+ free_object_addresses(immv_triggers);
+ }
+
+ /*
+ * Create triggers on incremental maintainable materialized view
+ * This argument should use 'dataQuery'. This needs to use a rewritten query,
+ * because a sublink in jointree is not supported by this function.
+ *
+ * This is performed before generating data because we have to wait
+ * concurrent transactions modifying a base table and then take a snapshot
+ * to see changes by these transactions to make sure a consistent view
+ * is created.
+ */
+ if (RelationIsIVM(matviewRel) && !skipData && !oldPopulated)
+ {
+ CreateIvmTriggersOnBaseTables(dataQuery, matviewOid);
+ CreateIndexOnIMMV(dataQuery, matviewRel);
+ }
+
/*
* Create the transient table that will receive the regenerated data. Lock
* it against access by any other process until commit (by which time it
@@ -326,8 +560,43 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
DestReceiver *dest;
dest = CreateTransientRelDestReceiver(OIDNewHeap);
- processed = refresh_matview_datafill(dest, dataQuery, queryString,
- is_create);
+
+ if (RelationIsIVM(matviewRel))
+ {
+ /*
+ * In READ COMMITTED, get and push the latest snapshot again to see the
+ * results of concurrent transactions committed after the current
+ * transaction started.
+ */
+ if (!IsolationUsesXactSnapshot())
+ PushActiveSnapshot(GetTransactionSnapshot());
+
+ /*
+ * If a concurrent transaction updated the view incrementally and was
+ * committed before we acquired the lock, the results of refresh_immv could
+ * be inconsistent. Therefore, we have to check the transaction ID of the
+ * most recent update of the view, and if this was in progress at the
+ * transaction start, raise an error to prevent anomalies.
+ */
+ if (!is_create)
+ {
+ FullTransactionId xid;
+
+ xid = getLastUpdateXid(matviewOid);
+ if (XidInMVCCSnapshot(XidFromFullTransactionId(xid), GetActiveSnapshot()))
+ ereport(ERROR,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("the materialized view is incrementally updated in concurrent transaction"),
+ errhint("The transaction might succeed if retried.")));
+ }
+ }
+
+ processed = refresh_matview_datafill(dest, dataQuery, NULL, NULL,
+ queryString, is_create);
+
+ /* Pop the original snapshot. */
+ if (RelationIsIVM(matviewRel) && !IsolationUsesXactSnapshot())
+ PopActiveSnapshot();
}
/* Make the matview match the newly generated data. */
@@ -402,6 +671,8 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
*/
static uint64
refresh_matview_datafill(DestReceiver *dest, Query *query,
+ QueryEnvironment *queryEnv,
+ TupleDesc *resultTupleDesc,
const char *queryString, bool is_create)
{
List *rewritten;
@@ -439,7 +710,7 @@ refresh_matview_datafill(DestReceiver *dest, Query *query,
/* Create a QueryDesc, redirecting output to our tuple receiver */
queryDesc = CreateQueryDesc(plan, queryString,
GetActiveSnapshot(), InvalidSnapshot,
- dest, NULL, NULL, 0);
+ dest, NULL, queryEnv ? queryEnv: NULL, 0);
/* call ExecutorStart to prepare the plan for execution */
ExecutorStart(queryDesc, 0);
@@ -449,6 +720,9 @@ refresh_matview_datafill(DestReceiver *dest, Query *query,
processed = queryDesc->estate->es_processed;
+ if (resultTupleDesc)
+ *resultTupleDesc = CreateTupleDescCopy(queryDesc->tupDesc);
+
/* and clean up */
ExecutorFinish(queryDesc);
ExecutorEnd(queryDesc);
@@ -967,3 +1241,1488 @@ CloseMatViewIncrementalMaintenance(void)
matview_maintenance_depth--;
Assert(matview_maintenance_depth >= 0);
}
+
+/*
+ * get_matview_query - get the Query from a matview's _RETURN rule.
+ */
+static Query *
+get_matview_query(Relation matviewRel)
+{
+ RewriteRule *rule;
+ List * actions;
+
+ /*
+ * Check that everything is correct for a refresh. Problems at this point
+ * are internal errors, so elog is sufficient.
+ */
+ if (matviewRel->rd_rel->relhasrules == false ||
+ matviewRel->rd_rules->numLocks < 1)
+ elog(ERROR,
+ "materialized view \"%s\" is missing rewrite information",
+ RelationGetRelationName(matviewRel));
+
+ if (matviewRel->rd_rules->numLocks > 1)
+ elog(ERROR,
+ "materialized view \"%s\" has too many rules",
+ RelationGetRelationName(matviewRel));
+
+ rule = matviewRel->rd_rules->rules[0];
+ if (rule->event != CMD_SELECT || !(rule->isInstead))
+ elog(ERROR,
+ "the rule for materialized view \"%s\" is not a SELECT INSTEAD OF rule",
+ RelationGetRelationName(matviewRel));
+
+ actions = rule->actions;
+ if (list_length(actions) != 1)
+ elog(ERROR,
+ "the rule for materialized view \"%s\" is not a single action",
+ RelationGetRelationName(matviewRel));
+
+ /*
+ * The stored query was rewritten at the time of the MV definition, but
+ * has not been scribbled on by the planner.
+ */
+ return linitial_node(Query, actions);
+}
+
+
+/* ----------------------------------------------------
+ * Incremental View Maintenance routines
+ * ---------------------------------------------------
+ */
+
+/*
+ * IVM_immediate_before
+ *
+ * IVM trigger function invoked before base table is modified. If this is
+ * invoked firstly in the same statement, we save the transaction id and the
+ * command id at that time.
+ */
+Datum
+IVM_immediate_before(PG_FUNCTION_ARGS)
+{
+ TriggerData *trigdata = (TriggerData *) fcinfo->context;
+ char *matviewOid_text = trigdata->tg_trigger->tgargs[0];
+ char *ex_lock_text = trigdata->tg_trigger->tgargs[1];
+ Oid matviewOid;
+ MV_TriggerHashEntry *entry;
+ bool found;
+ bool ex_lock;
+
+ matviewOid = DatumGetObjectId(DirectFunctionCall1(oidin, CStringGetDatum(matviewOid_text)));
+ ex_lock = DatumGetBool(DirectFunctionCall1(boolin, CStringGetDatum(ex_lock_text)));
+
+ /* If the view has more than one tables, we have to use an exclusive lock. */
+ if (ex_lock)
+ {
+ FullTransactionId xid;
+
+ /*
+ * Wait for concurrent transactions which update this materialized view at
+ * READ COMMITED. This is needed to see changes committed in other
+ * transactions. No wait and raise an error at REPEATABLE READ or
+ * SERIALIZABLE to prevent update anomalies of matviews.
+ * XXX: dead-lock is possible here.
+ */
+ if (!IsolationUsesXactSnapshot())
+ LockRelationOid(matviewOid, ExclusiveLock);
+ else if (!ConditionalLockRelationOid(matviewOid, ExclusiveLock))
+ {
+ /* try to throw error by name; relation could be deleted... */
+ char *relname = get_rel_name(matviewOid);
+
+ if (!relname)
+ ereport(ERROR,
+ (errcode(ERRCODE_LOCK_NOT_AVAILABLE),
+ errmsg("could not obtain lock on materialized view during incremental maintenance")));
+
+ ereport(ERROR,
+ (errcode(ERRCODE_LOCK_NOT_AVAILABLE),
+ errmsg("could not obtain lock on materialized view \"%s\" during incremental maintenance",
+ relname)));
+ }
+
+ /*
+ * Even if we can acquire an lock, a concurrent transaction could have
+ * updated the view incrementally and been committed before we acquired
+ * the lock. Therefore, we have to check the transaction ID of the most
+ * recent update of the view, and if this was in progress at the
+ * transaction start, raise an error to prevent anomalies.
+ */
+ xid = getLastUpdateXid(matviewOid);
+ if (XidInMVCCSnapshot(XidFromFullTransactionId(xid), GetTransactionSnapshot()))
+ ereport(ERROR,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("the materialized view is incrementally updated in concurrent transaction"),
+ errhint("The transaction might succeed if retried.")));
+ }
+ else
+ LockRelationOid(matviewOid, RowExclusiveLock);
+
+ /*
+ * On the first call initialize the hashtable
+ */
+ if (!mv_trigger_info)
+ mv_InitHashTables();
+
+ entry = (MV_TriggerHashEntry *) hash_search(mv_trigger_info,
+ (void *) &matviewOid,
+ HASH_ENTER, &found);
+
+ /* On the first BEFORE to update the view, initialize trigger data */
+ if (!found || entry->snapshot == InvalidSnapshot)
+ {
+ Snapshot snapshot;
+
+ /*
+ * Get a snapshot just before the table was modified for checking
+ * tuple visibility in the pre-update state of the table.
+ *
+ * In READ COMMITTED, use the latest snapshot again to see the
+ * results of concurrent transactions committed after the current
+ * transaction started.
+ */
+ if (IsolationUsesXactSnapshot())
+ snapshot = GetActiveSnapshot();
+ else
+ snapshot = GetTransactionSnapshot();
+
+ entry->matview_id = matviewOid;
+ entry->before_trig_count = 0;
+ entry->after_trig_count = 0;
+ entry->snapshot = RegisterSnapshot(snapshot);
+ entry->tables = NIL;
+ entry->has_old = false;
+ entry->has_new = false;
+
+ /*
+ * If this is the first table modifying query in the transaction,
+ * initialize the list of subxids.
+ */
+ if (!found)
+ entry->subxids = NIL;
+ }
+
+ entry->before_trig_count++;
+
+ return PointerGetDatum(NULL);
+}
+
+/*
+ * IVM_immediate_maintenance
+ *
+ * IVM trigger function invoked after base table is modified.
+ * For each table, tuplestores of transition tables are collected.
+ * and after the last modification
+ */
+Datum
+IVM_immediate_maintenance(PG_FUNCTION_ARGS)
+{
+ TriggerData *trigdata = (TriggerData *) fcinfo->context;
+ Relation rel;
+ Oid relid;
+ Oid matviewOid;
+ Query *query;
+ Query *rewritten = NULL;
+ char *matviewOid_text = trigdata->tg_trigger->tgargs[0];
+ Relation matviewRel;
+ int old_depth = matview_maintenance_depth;
+ SubTransactionId subxid;
+
+ Oid relowner;
+ Tuplestorestate *old_tuplestore = NULL;
+ Tuplestorestate *new_tuplestore = NULL;
+ DestReceiver *dest_new = NULL, *dest_old = NULL;
+ Oid save_userid;
+ int save_sec_context;
+ int save_nestlevel;
+
+ MV_TriggerHashEntry *entry;
+ MV_TriggerTable *table;
+ bool found;
+
+ ParseState *pstate;
+ QueryEnvironment *queryEnv = create_queryEnv();
+ MemoryContext oldcxt;
+ ListCell *lc;
+ int i;
+
+
+ /* Create a ParseState for rewriting the view definition query */
+ pstate = make_parsestate(NULL);
+ pstate->p_queryEnv = queryEnv;
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ rel = trigdata->tg_relation;
+ relid = rel->rd_id;
+
+ matviewOid = DatumGetObjectId(DirectFunctionCall1(oidin, CStringGetDatum(matviewOid_text)));
+
+ /*
+ * On the first call initialize the hashtable
+ */
+ if (!mv_trigger_info)
+ mv_InitHashTables();
+
+ /* get the entry for this materialized view */
+ entry = (MV_TriggerHashEntry *) hash_search(mv_trigger_info,
+ (void *) &matviewOid,
+ HASH_FIND, &found);
+ Assert (found && entry != NULL);
+ entry->after_trig_count++;
+
+ /* search the entry for the modified table and create new entry if not found */
+ found = false;
+ foreach(lc, entry->tables)
+ {
+ table = (MV_TriggerTable *) lfirst(lc);
+ if (table->table_id == relid)
+ {
+ found = true;
+ break;
+ }
+ }
+ if (!found)
+ {
+ oldcxt = MemoryContextSwitchTo(TopTransactionContext);
+
+ table = (MV_TriggerTable *) palloc0(sizeof(MV_TriggerTable));
+ table->table_id = relid;
+ table->old_tuplestores = NIL;
+ table->new_tuplestores = NIL;
+ table->rte_indexes = NIL;
+ table->slot = MakeSingleTupleTableSlot(RelationGetDescr(rel), table_slot_callbacks(rel));
+ /* We assume we have at least RowExclusiveLock on modified tables. */
+ table->rel = table_open(RelationGetRelid(rel), NoLock);
+ entry->tables = lappend(entry->tables, table);
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ /* Save the transition tables and make a request to not free immediately */
+ if (trigdata->tg_oldtable)
+ {
+ oldcxt = MemoryContextSwitchTo(TopTransactionContext);
+ table->old_tuplestores = lappend(table->old_tuplestores, trigdata->tg_oldtable);
+ entry->has_old = true;
+ MemoryContextSwitchTo(oldcxt);
+ }
+ if (trigdata->tg_newtable)
+ {
+ oldcxt = MemoryContextSwitchTo(TopTransactionContext);
+ table->new_tuplestores = lappend(table->new_tuplestores, trigdata->tg_newtable);
+ entry->has_new = true;
+ MemoryContextSwitchTo(oldcxt);
+ }
+ if (entry->has_new || entry->has_old)
+ {
+ CmdType cmd;
+
+ if (TRIGGER_FIRED_BY_INSERT(trigdata->tg_event))
+ cmd = CMD_INSERT;
+ else if (TRIGGER_FIRED_BY_DELETE(trigdata->tg_event))
+ cmd = CMD_DELETE;
+ else if (TRIGGER_FIRED_BY_UPDATE(trigdata->tg_event))
+ cmd = CMD_UPDATE;
+ else
+ elog(ERROR,"unsupported trigger type");
+
+ /* Prolong lifespan of transition tables to the end of the last AFTER trigger */
+ SetTransitionTablePreserved(relid, cmd);
+ }
+
+
+ /* If this is not the last AFTER trigger call, immediately exit. */
+ Assert (entry->before_trig_count >= entry->after_trig_count);
+ if (entry->before_trig_count != entry->after_trig_count)
+ return PointerGetDatum(NULL);
+
+ /*
+ * If this is the last AFTER trigger call, continue and update the view.
+ */
+
+ /* record the subxid that updated the view incrementally */
+ subxid = GetCurrentSubTransactionId();
+ if (!list_member_xid(entry->subxids, subxid))
+ {
+ oldcxt = MemoryContextSwitchTo(TopTransactionContext);
+ entry->subxids = lappend_xid(entry->subxids, subxid);
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ /*
+ * Advance command counter to make the updated base table rows locally
+ * visible.
+ */
+ CommandCounterIncrement();
+
+ matviewRel = table_open(matviewOid, NoLock);
+
+ /* Make sure it is a materialized view. */
+ Assert(matviewRel->rd_rel->relkind == RELKIND_MATVIEW);
+
+ /*
+ * In READ COMMITTED, get and push the latest snapshot again to see the
+ * results of concurrent transactions committed after the current
+ * transaction started.
+ */
+ if (!IsolationUsesXactSnapshot())
+ PushActiveSnapshot(GetTransactionSnapshot());
+
+ /*
+ * Check for active uses of the relation in the current transaction, such
+ * as open scans.
+ *
+ * NB: We count on this to protect us against problems with refreshing the
+ * data using TABLE_INSERT_FROZEN.
+ */
+ CheckTableNotInUse(matviewRel, "refresh a materialized view incrementally");
+
+ /*
+ * Switch to the owner's userid, so that any functions are run as that
+ * user. Also arrange to make GUC variable changes local to this command.
+ * We will switch modes when we are about to execute user code.
+ */
+ relowner = matviewRel->rd_rel->relowner;
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_RESTRICTED_OPERATION);
+ save_nestlevel = NewGUCNestLevel();
+
+ /* get view query*/
+ query = get_matview_query(matviewRel);
+
+ /*
+ * When a base table is truncated, the view content will be empty if the
+ * view definition query does not contain an aggregate without a GROUP clause.
+ * Therefore, such views can be truncated.
+ */
+ if (TRIGGER_FIRED_BY_TRUNCATE(trigdata->tg_event))
+ {
+ ExecuteTruncateGuts(list_make1(matviewRel), list_make1_oid(matviewOid),
+ NIL, DROP_RESTRICT, false, false);
+
+ /* Clean up hash entry and delete tuplestores */
+ clean_up_IVM_hash_entry(entry, false, InvalidSubTransactionId);
+
+ /* Pop the original snapshot. */
+ if (!IsolationUsesXactSnapshot())
+ PopActiveSnapshot();
+
+ table_close(matviewRel, NoLock);
+
+ /* Roll back any GUC changes */
+ AtEOXact_GUC(false, save_nestlevel);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+
+ return PointerGetDatum(NULL);
+ }
+
+ /*
+ * rewrite query for calculating deltas
+ */
+
+ rewritten = copyObject(query);
+
+ /* Replace resnames in a target list with materialized view's attnames */
+ i = 0;
+ foreach (lc, rewritten->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, i);
+ char *resname = NameStr(attr->attname);
+
+ tle->resname = pstrdup(resname);
+ i++;
+ }
+
+ /* Set all tables in the query to pre-update state */
+ rewritten = rewrite_query_for_preupdate_state(rewritten, entry->tables,
+ pstate, matviewOid);
+ /* Rewrite for counting duplicated tuples */
+ rewritten = rewrite_query_for_counting(rewritten, pstate);
+
+ /* Create tuplestores to store view deltas */
+ if (entry->has_old)
+ {
+ oldcxt = MemoryContextSwitchTo(TopTransactionContext);
+
+ old_tuplestore = tuplestore_begin_heap(false, false, work_mem);
+ dest_old = CreateDestReceiver(DestTuplestore);
+ SetTuplestoreDestReceiverParams(dest_old,
+ old_tuplestore,
+ TopTransactionContext,
+ false,
+ NULL,
+ NULL);
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+ if (entry->has_new)
+ {
+ oldcxt = MemoryContextSwitchTo(TopTransactionContext);
+
+ new_tuplestore = tuplestore_begin_heap(false, false, work_mem);
+ dest_new = CreateDestReceiver(DestTuplestore);
+ SetTuplestoreDestReceiverParams(dest_new,
+ new_tuplestore,
+ TopTransactionContext,
+ false,
+ NULL,
+ NULL);
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ /* for all modified tables */
+ foreach(lc, entry->tables)
+ {
+ ListCell *lc2;
+
+ table = (MV_TriggerTable *) lfirst(lc);
+
+ /* loop for self-join */
+ foreach(lc2, table->rte_indexes)
+ {
+ int rte_index = lfirst_int(lc2);
+ TupleDesc tupdesc_old;
+ TupleDesc tupdesc_new;
+
+ /* calculate delta tables */
+ calc_delta(table, rte_index, rewritten, dest_old, dest_new,
+ &tupdesc_old, &tupdesc_new, queryEnv);
+
+ /* Set the table in the query to post-update state */
+ rewritten = rewrite_query_for_postupdate_state(rewritten, table, rte_index);
+
+ PG_TRY();
+ {
+ /* apply the delta tables to the materialized view */
+ apply_delta(matviewOid, old_tuplestore, new_tuplestore,
+ tupdesc_old, tupdesc_new, query);
+ }
+ PG_CATCH();
+ {
+ matview_maintenance_depth = old_depth;
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /* clear view delta tuplestores */
+ if (old_tuplestore)
+ tuplestore_clear(old_tuplestore);
+ if (new_tuplestore)
+ tuplestore_clear(new_tuplestore);
+ }
+ }
+
+ /* Clean up hash entry and delete tuplestores */
+ clean_up_IVM_hash_entry(entry, false, InvalidSubTransactionId);
+ if (old_tuplestore)
+ {
+ dest_old->rDestroy(dest_old);
+ tuplestore_end(old_tuplestore);
+ }
+ if (new_tuplestore)
+ {
+ dest_new->rDestroy(dest_new);
+ tuplestore_end(new_tuplestore);
+ }
+
+ /* Pop the original snapshot. */
+ if (!IsolationUsesXactSnapshot())
+ PopActiveSnapshot();
+
+ table_close(matviewRel, NoLock);
+
+ /* Roll back any GUC changes */
+ AtEOXact_GUC(false, save_nestlevel);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+
+ return PointerGetDatum(NULL);
+}
+
+/*
+ * rewrite_query_for_preupdate_state
+ *
+ * Rewrite the query so that base tables' RTEs will represent "pre-update"
+ * state of tables. This is necessary to calculate view delta after multiple
+ * tables are modified.
+ */
+static Query*
+rewrite_query_for_preupdate_state(Query *query, List *tables,
+ ParseState *pstate, Oid matviewid)
+{
+ ListCell *lc;
+ int num_rte = list_length(query->rtable);
+ int i;
+
+
+ /* register delta ENRs */
+ register_delta_ENRs(pstate, query, tables);
+
+ /* XXX: Is necessary? Is this right timing? */
+ AcquireRewriteLocks(query, true, false);
+
+ i = 1;
+ foreach(lc, query->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc);
+
+ ListCell *lc2;
+ foreach(lc2, tables)
+ {
+ MV_TriggerTable *table = (MV_TriggerTable *) lfirst(lc2);
+ /*
+ * if the modified table is found then replace the original RTE with
+ * "pre-state" RTE and append its index to the list.
+ */
+ if (r->relid == table->table_id)
+ {
+ List *securityQuals;
+ List *withCheckOptions;
+ bool hasRowSecurity;
+ bool hasSubLinks;
+
+ RangeTblEntry *rte_pre = get_prestate_rte(r, table, pstate->p_queryEnv, matviewid);
+
+ /*
+ * Set a row security poslicies of the modified table to the subquery RTE which
+ * represents the pre-update state of the table.
+ */
+ get_row_security_policies(query, table->original_rte, i,
+ &securityQuals, &withCheckOptions,
+ &hasRowSecurity, &hasSubLinks);
+
+ if (hasRowSecurity)
+ {
+ query->hasRowSecurity = true;
+ rte_pre->security_barrier = true;
+ }
+ if (hasSubLinks)
+ query->hasSubLinks = true;
+
+ rte_pre->securityQuals = securityQuals;
+ lfirst(lc) = rte_pre;
+
+ table->rte_indexes = lappend_int(table->rte_indexes, i);
+ break;
+ }
+ }
+
+ /* finish the loop if we processed all RTE included in the original query */
+ if (i++ >= num_rte)
+ break;
+ }
+
+ return query;
+}
+
+/*
+ * register_delta_ENRs
+ *
+ * For all modified tables, make ENRs for their transition tables
+ * and register them to the queryEnv. ENR's RTEs are also appended
+ * into the list in query tree.
+ */
+static void
+register_delta_ENRs(ParseState *pstate, Query *query, List *tables)
+{
+ QueryEnvironment *queryEnv = pstate->p_queryEnv;
+ ListCell *lc;
+ RangeTblEntry *rte;
+
+ foreach(lc, tables)
+ {
+ MV_TriggerTable *table = (MV_TriggerTable *) lfirst(lc);
+ ListCell *lc2;
+ int count;
+
+ count = 0;
+ foreach(lc2, table->old_tuplestores)
+ {
+ Tuplestorestate *oldtable = (Tuplestorestate *) lfirst(lc2);
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+ ParseNamespaceItem *nsitem;
+
+ enr->md.name = make_delta_enr_name("old", table->table_id, count);
+ enr->md.reliddesc = table->table_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(oldtable);
+ enr->reldata = oldtable;
+ register_ENR(queryEnv, enr);
+
+ nsitem = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ rte = nsitem->p_rte;
+
+ query->rtable = lappend(query->rtable, rte);
+
+ count++;
+ }
+
+ count = 0;
+ foreach(lc2, table->new_tuplestores)
+ {
+ Tuplestorestate *newtable = (Tuplestorestate *) lfirst(lc2);
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+ ParseNamespaceItem *nsitem;
+
+ enr->md.name = make_delta_enr_name("new", table->table_id, count);
+ enr->md.reliddesc = table->table_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(newtable);
+ enr->reldata = newtable;
+ register_ENR(queryEnv, enr);
+
+ nsitem = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ rte = nsitem->p_rte;
+
+ query->rtable = lappend(query->rtable, rte);
+
+ count++;
+ }
+ }
+}
+
+#define DatumGetItemPointer(X) ((ItemPointer) DatumGetPointer(X))
+#define PG_GETARG_ITEMPOINTER(n) DatumGetItemPointer(PG_GETARG_DATUM(n))
+
+/*
+ * ivm_visible_in_prestate
+ *
+ * Check visibility of a tuple specified by the tableoid and item pointer
+ * using the snapshot taken just before the table was modified.
+ */
+Datum
+ivm_visible_in_prestate(PG_FUNCTION_ARGS)
+{
+ Oid tableoid = PG_GETARG_OID(0);
+ ItemPointer itemPtr = PG_GETARG_ITEMPOINTER(1);
+ Oid matviewOid = PG_GETARG_OID(2);
+ MV_TriggerHashEntry *entry;
+ MV_TriggerTable *table = NULL;
+ ListCell *lc;
+ bool found;
+ bool result;
+
+ if (!in_delta_calculation)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("ivm_visible_in_prestate can be called only in delta calculation")));
+
+ entry = (MV_TriggerHashEntry *) hash_search(mv_trigger_info,
+ (void *) &matviewOid,
+ HASH_FIND, &found);
+ Assert (found && entry != NULL);
+
+ foreach(lc, entry->tables)
+ {
+ table = (MV_TriggerTable *) lfirst(lc);
+ if (table->table_id == tableoid)
+ break;
+ }
+
+ Assert (table != NULL);
+
+ result = table_tuple_fetch_row_version(table->rel, itemPtr, entry->snapshot, table->slot);
+
+ PG_RETURN_BOOL(result);
+}
+
+/*
+ * get_prestate_rte
+ *
+ * Rewrite RTE of the modified table to a subquery which represents
+ * "pre-state" table. The original RTE is saved in table->rte_original.
+ */
+static RangeTblEntry*
+get_prestate_rte(RangeTblEntry *rte, MV_TriggerTable *table,
+ QueryEnvironment *queryEnv, Oid matviewid)
+{
+ StringInfoData str;
+ RawStmt *raw;
+ Query *subquery;
+ ParseState *pstate;
+ char *relname;
+ static char *subquery_tl;
+ int i;
+
+ pstate = make_parsestate(NULL);
+ pstate->p_queryEnv = queryEnv;
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ relname = quote_qualified_identifier(
+ get_namespace_name(RelationGetNamespace(table->rel)),
+ RelationGetRelationName(table->rel));
+
+ subquery_tl = make_subquery_targetlist_from_table(table);
+
+ /*
+ * Filtering inserted row using the snapshot taken before the table
+ * is modified. ctid is required for maintaining outer join views.
+ */
+ initStringInfo(&str);
+ appendStringInfo(&str,
+ "SELECT %s FROM %s t"
+ " WHERE pg_catalog.ivm_visible_in_prestate(t.tableoid, t.ctid ,%d::pg_catalog.oid)",
+ subquery_tl, relname, matviewid);
+
+ /*
+ * Re-add rows deleted from the old transition tables, excluding those
+ * also present in the new transition tables.
+ */
+ if (list_length(table->old_tuplestores) > 0)
+ {
+ appendStringInfo(&str," UNION ALL SELECT %s FROM (",
+ subquery_tl);
+
+ for (i = 0; i < list_length(table->old_tuplestores); i++)
+ {
+ if (i != 0)
+ appendStringInfo(&str, " UNION ALL ");
+ appendStringInfo(&str," TABLE %s",
+ make_delta_enr_name("old", table->table_id, i));
+ }
+ for (i = 0; i < list_length(table->new_tuplestores); i++)
+ {
+ appendStringInfo(&str, " EXCEPT ALL ");
+ appendStringInfo(&str," TABLE %s",
+ make_delta_enr_name("new", table->table_id, i));
+ }
+ appendStringInfo(&str,")");
+ }
+
+ /* Get a subquery representing pre-state of the table */
+ raw = (RawStmt*)linitial(raw_parser(str.data, RAW_PARSE_DEFAULT));
+ subquery = transformStmt(pstate, raw->stmt);
+
+ /* save the original RTE */
+ table->original_rte = copyObject(rte);
+
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = subquery;
+ rte->security_barrier = false;
+
+ /* Clear fields that should not be set in a subquery RTE */
+ rte->relid = InvalidOid;
+ rte->relkind = 0;
+ rte->rellockmode = 0;
+ rte->tablesample = NULL;
+ rte->perminfoindex = 0; /* no permission checking for this RTE */
+ rte->inh = false; /* must not be set for a subquery */
+
+ return rte;
+}
+
+/*
+ * make_subquery_targetlist_from_table
+ *
+ * Make a targetlist string of a subquery representing a delta table or a
+ * pre-update state table. This subquery substitutes a modified table RTE
+ * in the view definition query during view maintenance. In the targetlist,
+ * column names appear in order of the table definition. However, for
+ * attribute numbers of vars in the query tree to reference columns of the
+ * subquery correctly even though the table has a dropped column, put "null"
+ * as a dummy value at the position of a dropped column.
+ *
+ * We would also able to walk the query tree to rewrite varattnos, but
+ * crafting targetlist is more simple and reasonable.
+ */
+static char*
+make_subquery_targetlist_from_table(MV_TriggerTable *table)
+{
+ StringInfoData str;
+ TupleDesc tupdesc;
+ int i;
+
+ tupdesc = RelationGetDescr(table->rel);
+ initStringInfo(&str);
+ for (i = 0; i < tupdesc->natts; i++)
+ {
+ Form_pg_attribute attr = TupleDescAttr(tupdesc, i);
+
+ if (i > 0)
+ appendStringInfo(&str, ", ");
+
+ if (attr->attisdropped)
+ appendStringInfo(&str, "null");
+ else
+ appendStringInfo(&str, "%s", quote_identifier(NameStr(attr->attname)));
+ }
+
+ return str.data;
+}
+
+/*
+ * make_delta_enr_name
+ *
+ * Make a name for ENR of a transition table from the base table's oid.
+ * prefix will be "new" or "old" depending on its transition table kind..
+ */
+static char*
+make_delta_enr_name(const char *prefix, Oid relid, int count)
+{
+ char buf[NAMEDATALEN];
+ char *name;
+
+ snprintf(buf, NAMEDATALEN, "__ivm_%s_%u_%u", prefix, relid, count);
+ name = pstrdup(buf);
+
+ return name;
+}
+
+/*
+ * makeDeltaTable
+ *
+ * Make a RTE representing a delta of the specified table.
+ */
+static RangeTblEntry*
+makeDeltaTable(RangeTblEntry *rte, MV_TriggerTable *table,
+ bool is_new, QueryEnvironment *queryEnv)
+{
+ StringInfoData str;
+ ParseState *pstate;
+ RawStmt *raw;
+ Query *sub;
+ int i;
+
+ const char *prefix_union = is_new ? "new" : "old";
+ const char *prefix_except = is_new ? "old" : "new";
+ int num_union = is_new ? list_length(table->new_tuplestores) : list_length(table->old_tuplestores);
+ int num_except = is_new ? list_length(table->old_tuplestores) : list_length(table->new_tuplestores);
+
+ /* the previous RTE must be a subquery which represents "pre-state" table */
+ Assert(rte->rtekind == RTE_SUBQUERY);
+
+ /* Create a ParseState for rewriting the view definition query */
+ pstate = make_parsestate(NULL);
+ pstate->p_queryEnv = queryEnv;
+ pstate->p_expr_kind = EXPR_KIND_NONE;
+
+ initStringInfo(&str);
+ appendStringInfo(&str,
+ " SELECT %s FROM ( ",
+ make_subquery_targetlist_from_table(table));
+
+ for (i = 0; i < num_union; i++)
+ {
+ if (i > 0)
+ appendStringInfo(&str, " UNION ALL ");
+
+ appendStringInfo(&str,
+ " TABLE %s",
+ make_delta_enr_name(prefix_union, table->table_id, i));
+ }
+ for (i = 0; i < num_except; i++)
+ {
+ appendStringInfo(&str, " EXCEPT ALL ");
+
+ appendStringInfo(&str,
+ " TABLE %s",
+ make_delta_enr_name(prefix_except, table->table_id, i));
+ }
+ appendStringInfo(&str,")");
+
+ raw = (RawStmt*)linitial(raw_parser(str.data, RAW_PARSE_DEFAULT));
+ sub = transformStmt(pstate, raw->stmt);
+
+ /*
+ * Update the subquery so that it represent the combined transition
+ * table. Note that we leave the security_barrier and securityQuals
+ * fields so that the subquery relation can be protected by the RLS
+ * policy as same as the modified table.
+ */
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = sub;
+
+ return rte;
+}
+
+/*
+ * rewrite_query_for_counting
+ *
+ * Rewrite query for counting duplicated tuples.
+ */
+static Query *
+rewrite_query_for_counting(Query *query, ParseState *pstate)
+{
+ TargetEntry *tle_count;
+ FuncCall *fn;
+ Node *node;
+
+ /* Add count(*) for counting distinct tuples in views */
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+ if (!query->groupClause && !query->hasAggs)
+ query->groupClause = transformDistinctClause(NULL, &query->targetList, query->sortClause, false);
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle_count = makeTargetEntry((Expr *) node,
+ list_length(query->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ query->targetList = lappend(query->targetList, tle_count);
+ query->hasAggs = true;
+
+ return query;
+}
+
+/*
+ * calc_delta
+ *
+ * Calculate view deltas generated under the modification of a table specified
+ * by the RTE index.
+ */
+static void
+calc_delta(MV_TriggerTable *table, int rte_index, Query *query,
+ DestReceiver *dest_old, DestReceiver *dest_new,
+ TupleDesc *tupdesc_old, TupleDesc *tupdesc_new,
+ QueryEnvironment *queryEnv)
+{
+ ListCell *lc = list_nth_cell(query->rtable, rte_index - 1);
+ RangeTblEntry *rte = (RangeTblEntry *) lfirst(lc);
+
+ in_delta_calculation = true;
+
+ /* Generate old delta */
+ if (dest_old && list_length(table->old_tuplestores) > 0)
+ {
+ /* Replace the modified table with the old delta table and calculate the old view delta. */
+ lfirst(lc) = makeDeltaTable(rte, table, false, queryEnv);
+ refresh_matview_datafill(dest_old, query, queryEnv, tupdesc_old, "", false);
+ }
+
+ /* Generate new delta */
+ if (dest_new && list_length(table->new_tuplestores) > 0)
+ {
+ /* Replace the modified table with the new delta table and calculate the new view delta*/
+ lfirst(lc) = makeDeltaTable(rte, table, true, queryEnv);
+ refresh_matview_datafill(dest_new, query, queryEnv, tupdesc_new, "", false);
+ }
+
+ in_delta_calculation = false;
+}
+
+/*
+ * rewrite_query_for_postupdate_state
+ *
+ * Rewrite the query so that the specified base table's RTEs will represent
+ * "post-update" state of tables. This is called after the view delta
+ * calculation due to changes on this table finishes.
+ */
+static Query*
+rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte_index)
+{
+ ListCell *lc = list_nth_cell(query->rtable, rte_index - 1);
+
+ /* Retore the original RTE */
+ lfirst(lc) = table->original_rte;
+
+ return query;
+}
+
+/*
+ * apply_delta
+ *
+ * Apply deltas to the materialized view. In outer join cases, this requires
+ * the view maintenance graph.
+ */
+static void
+apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
+ TupleDesc tupdesc_old, TupleDesc tupdesc_new,
+ Query *query)
+{
+ StringInfoData querybuf;
+ StringInfoData target_list_buf;
+ Relation matviewRel;
+ char *matviewname;
+ ListCell *lc;
+ int i;
+ List *keys = NIL;
+
+
+ /*
+ * get names of the materialized view and delta tables
+ */
+
+ matviewRel = table_open(matviewOid, NoLock);
+ matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)),
+ RelationGetRelationName(matviewRel));
+
+ /*
+ * Build parts of the maintenance queries
+ */
+
+ initStringInfo(&querybuf);
+ initStringInfo(&target_list_buf);
+
+ /* build string of target list */
+ for (i = 0; i < matviewRel->rd_att->natts; i++)
+ {
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, i);
+ char *resname = NameStr(attr->attname);
+
+ if (i != 0)
+ appendStringInfo(&target_list_buf, ", ");
+ appendStringInfo(&target_list_buf, "%s", quote_qualified_identifier(NULL, resname));
+ }
+
+ i = 0;
+ foreach (lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, i);
+
+ i++;
+
+ if (tle->resjunk)
+ continue;
+
+ keys = lappend(keys, attr);
+ }
+
+ /* Start maintaining the materialized view. */
+ OpenMatViewIncrementalMaintenance();
+
+ /* Open SPI context. */
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ /* For tuple deletion */
+ if (old_tuplestores && tuplestore_tuple_count(old_tuplestores) > 0)
+ {
+ EphemeralNamedRelation enr = palloc(sizeof(EphemeralNamedRelationData));
+ int rc;
+
+ /* convert tuplestores to ENR, and register for SPI */
+ enr->md.name = pstrdup(OLD_DELTA_ENRNAME);
+ enr->md.reliddesc = InvalidOid;
+ enr->md.tupdesc = tupdesc_old;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(old_tuplestores);
+ enr->reldata = old_tuplestores;
+
+ rc = SPI_register_relation(enr);
+ if (rc != SPI_OK_REL_REGISTER)
+ elog(ERROR, "SPI_register failed");
+
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+
+ }
+ /* For tuple insertion */
+ if (new_tuplestores && tuplestore_tuple_count(new_tuplestores) > 0)
+ {
+ EphemeralNamedRelation enr = palloc(sizeof(EphemeralNamedRelationData));
+ int rc;
+
+ /* convert tuplestores to ENR, and register for SPI */
+ enr->md.name = pstrdup(NEW_DELTA_ENRNAME);
+ enr->md.reliddesc = InvalidOid;
+ enr->md.tupdesc = tupdesc_new;;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(new_tuplestores);
+ enr->reldata = new_tuplestores;
+
+ rc = SPI_register_relation(enr);
+ if (rc != SPI_OK_REL_REGISTER)
+ elog(ERROR, "SPI_register failed");
+
+ /* apply new delta */
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ }
+
+ /* We're done maintaining the materialized view. */
+ CloseMatViewIncrementalMaintenance();
+
+ table_close(matviewRel, NoLock);
+
+ /* Close SPI context. */
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+}
+
+/*
+ * apply_old_delta
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is not required.
+ */
+static void
+apply_old_delta(const char *matviewname, const char *deltaname_old,
+ List *keys)
+{
+ StringInfoData querybuf;
+ StringInfoData keysbuf;
+ char *match_cond;
+ ListCell *lc;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&keysbuf);
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&keysbuf, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&keysbuf, ", ");
+ }
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "DELETE FROM %s WHERE ctid IN ("
+ "SELECT tid FROM (SELECT pg_catalog.row_number() over (partition by %s) AS \"__ivm_row_number__\","
+ "mv.ctid AS tid,"
+ "diff.\"__ivm_count__\""
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s) v "
+ "WHERE v.\"__ivm_row_number__\" OPERATOR(pg_catalog.<=) v.\"__ivm_count__\")",
+ matviewname,
+ keysbuf.data,
+ matviewname, deltaname_old,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
+/*
+ * apply_new_delta
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is not required.
+ */
+static void
+apply_new_delta(const char *matviewname, const char *deltaname_new,
+ StringInfo target_list)
+{
+ StringInfoData querybuf;
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "INSERT INTO %s (%s) SELECT %s FROM ("
+ "SELECT diff.*, pg_catalog.generate_series(1, diff.\"__ivm_count__\")"
+ " AS __ivm_generate_series__ "
+ "FROM %s AS diff) AS v",
+ matviewname, target_list->data, target_list->data,
+ deltaname_new);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
+/*
+ * get_matching_condition_string
+ *
+ * Build a predicate string for looking for a tuple with given keys.
+ */
+static char *
+get_matching_condition_string(List *keys)
+{
+ StringInfoData match_cond;
+ ListCell *lc;
+
+ /* If there is no key columns, the condition is always true. */
+ if (keys == NIL)
+ return "true";
+
+ initStringInfo(&match_cond);
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ char *mv_resname = quote_qualified_identifier("mv", resname);
+ char *diff_resname = quote_qualified_identifier("diff", resname);
+ Oid typid = attr->atttypid;
+
+ /* Considering NULL values, we can not use simple = operator. */
+ appendStringInfo(&match_cond, "(");
+ generate_equal(&match_cond, typid, mv_resname, diff_resname);
+ appendStringInfo(&match_cond, " OR (%s IS NULL AND %s IS NULL))",
+ mv_resname, diff_resname);
+
+ if (lnext(keys, lc))
+ appendStringInfo(&match_cond, " AND ");
+ }
+
+ return match_cond.data;
+}
+
+/*
+ * generate_equals
+ *
+ * Generate an equality clause using given operands' default equality
+ * operator.
+ */
+static void
+generate_equal(StringInfo querybuf, Oid opttype,
+ const char *leftop, const char *rightop)
+{
+ TypeCacheEntry *typentry;
+
+ typentry = lookup_type_cache(opttype, TYPECACHE_EQ_OPR);
+ if (!OidIsValid(typentry->eq_opr))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_FUNCTION),
+ errmsg("could not identify an equality operator for type %s",
+ format_type_be_qualified(opttype))));
+
+ generate_operator_clause(querybuf,
+ leftop, opttype,
+ typentry->eq_opr,
+ rightop, opttype);
+}
+
+/*
+ * mv_InitHashTables
+ */
+static void
+mv_InitHashTables(void)
+{
+ HASHCTL ctl;
+
+ memset(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(Oid);
+ ctl.entrysize = sizeof(MV_TriggerHashEntry);
+ mv_trigger_info = hash_create("MV trigger info",
+ MV_INIT_QUERYHASHSIZE,
+ &ctl, HASH_ELEM | HASH_BLOBS);
+
+ memset(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(Oid);
+ ctl.entrysize = sizeof(DroppedImmvInfo);
+ dropped_immv = hash_create("Dropped IMMVs", 16,
+ &ctl, HASH_ELEM | HASH_BLOBS);
+}
+
+/*
+ * AtAbort_IVM
+ *
+ * Clean up hash entries for all materialized views. This is called at
+ * (sub-)transaction abort. When the top-level transaction is aborted,
+ * InvalidSubTransactionId is set to subxid.
+ *
+ * Also, remove dropped IMMV information if it is aborted.
+ */
+void
+AtAbort_IVM(SubTransactionId subxid)
+{
+ HASH_SEQ_STATUS seq;
+
+ if (mv_trigger_info)
+ {
+ MV_TriggerHashEntry *entry;
+ hash_seq_init(&seq, mv_trigger_info);
+ while ((entry = hash_seq_search(&seq)) != NULL)
+ clean_up_IVM_hash_entry(entry, true, subxid);
+ }
+
+ if (dropped_immv)
+ {
+ DroppedImmvInfo *entry;
+ hash_seq_init(&seq, dropped_immv);
+ while ((entry = hash_seq_search(&seq)) != NULL)
+ {
+ if (subxid == InvalidSubTransactionId || subxid == entry->subxid)
+ hash_search(dropped_immv, (void *) &entry->immv_oid, HASH_REMOVE, NULL);
+ }
+ }
+
+ in_delta_calculation = false;
+}
+
+/*
+ * AtPreCommit_IVM
+ *
+ * Record the transaction ID that updated the view incrementally.
+ * Also, remove the entry for dropped IMMVs.
+ */
+void
+AtPreCommit_IVM(void)
+{
+ HASH_SEQ_STATUS seq;
+
+ if (mv_trigger_info)
+ {
+ MV_TriggerHashEntry *entry;
+
+ /*
+ * Record the transaction ID that udpate the view incrementally,
+ * and perform the final clean up of the entry.
+ */
+ hash_seq_init(&seq, mv_trigger_info);
+ while ((entry = hash_seq_search(&seq)) != NULL)
+ {
+ setLastUpdateXid(entry->matview_id, GetTopFullTransactionId());
+ hash_search(mv_trigger_info, (void *) &entry->matview_id, HASH_REMOVE, NULL);
+ }
+ }
+
+ if (dropped_immv)
+ {
+ DroppedImmvInfo *entry;
+
+ LWLockAcquire(IvmControlLock, LW_EXCLUSIVE);
+ hash_seq_init(&seq, dropped_immv);
+ while ((entry = hash_seq_search(&seq)) != NULL)
+ {
+ hash_search(LastIvmUpdateHash, (void *) &entry->immv_oid, HASH_REMOVE, NULL);
+ hash_search(dropped_immv, (void *) &entry->immv_oid, HASH_REMOVE, NULL);
+ }
+ LWLockRelease(IvmControlLock);
+ }
+
+ in_delta_calculation = false;
+}
+
+/*
+ * clean_up_IVM_hash_entry
+ *
+ * Clean up tuple stores and hash entries for a materialized view after its
+ * maintenance finished. This is called at the end of table modifying query
+ * or (sub-)transaction abort. When the top-level transaction is aborted,
+ * InvalidSubTransactionId is set to subxid.
+ */
+static void
+clean_up_IVM_hash_entry(MV_TriggerHashEntry *entry, bool is_abort,
+ SubTransactionId subxid)
+{
+ bool found;
+ ListCell *lc;
+
+ /* clean up tuple stores */
+ foreach(lc, entry->tables)
+ {
+ MV_TriggerTable *table = (MV_TriggerTable *) lfirst(lc);
+
+ list_free(table->old_tuplestores);
+ list_free(table->new_tuplestores);
+ if (!is_abort)
+ {
+ ExecDropSingleTupleTableSlot(table->slot);
+ table_close(table->rel, NoLock);
+ }
+ }
+ list_free(entry->tables);
+ entry->tables = NIL;
+
+ if (is_abort)
+ {
+ bool remove_entry = false;
+
+ /*
+ * When the top-level transaction is aborted, remove all subxids.
+ * When a sub-transaction is aborted, remove only its subxid.
+ */
+ if (subxid == InvalidSubTransactionId)
+ remove_entry = true;
+ else
+ {
+ foreach(lc, entry->subxids)
+ {
+ if (lfirst_xid(lc) == subxid)
+ {
+ entry->subxids = list_delete_cell(entry->subxids, lc);
+ break;
+ }
+ }
+
+ /*
+ * If all the subxid are removed, it means that the view was not
+ * updated at all in this transaction.
+ */
+ if (list_length(entry->subxids) == 0)
+ remove_entry = true;
+ }
+
+ /*
+ * Remove entries of not updated views from the hash table.
+ */
+ if (remove_entry)
+ hash_search(mv_trigger_info, (void *) &entry->matview_id, HASH_REMOVE, &found);
+ }
+ else
+ {
+ /* When the query sucsessully finished, unregister the snapshot */
+ UnregisterSnapshot(entry->snapshot);
+ }
+
+ entry->snapshot = InvalidSnapshot;
+}
+
+/*
+ * setLastUpdateXid
+ *
+ * Store the transaction ID that updated the view incremenally.
+ */
+static void
+setLastUpdateXid(Oid immv_oid, FullTransactionId xid)
+{
+ LastIvmUpdateEntry *entry;
+
+ LWLockAcquire(IvmControlLock, LW_EXCLUSIVE);
+ entry = (LastIvmUpdateEntry *) hash_search(LastIvmUpdateHash, (void *) &immv_oid, HASH_ENTER, NULL);
+ entry->oid = immv_oid;
+ entry->last_ivm_update = xid;
+ LWLockRelease(IvmControlLock);
+}
+
+/*
+ * getLastUpdateXid
+ *
+ * Get the most recent transaction ID that updated the view incrementally.
+ */
+static FullTransactionId
+getLastUpdateXid(Oid immv_oid)
+{
+ FullTransactionId xid = InvalidFullTransactionId;
+ LastIvmUpdateEntry *entry;
+ bool found;
+
+ LWLockAcquire(IvmControlLock, LW_SHARED);
+ entry = (LastIvmUpdateEntry *) hash_search(LastIvmUpdateHash, (void *) &immv_oid, HASH_FIND, &found);
+ LWLockRelease(IvmControlLock);
+ if (found)
+ xid = entry->last_ivm_update;
+
+ return xid;
+}
+
+void
+removeImmv(Oid immv_oid)
+{
+ DroppedImmvInfo *entry;
+
+ if (!dropped_immv)
+ mv_InitHashTables();
+
+ entry = (DroppedImmvInfo *) hash_search(dropped_immv, (void *) &immv_oid, HASH_ENTER, NULL);
+ entry->subxid = GetCurrentSubTransactionId();
+}
+
+/*
+ * isIvmName
+ *
+ * Check if this is a IVM hidden column from the name.
+ */
+bool
+isIvmName(const char *s)
+{
+ if (s)
+ return (strncmp(s, "__ivm_", 6) == 0);
+ return false;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index a1845240a98..99bc709959a 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -62,6 +62,7 @@
#include "commands/defrem.h"
#include "commands/event_trigger.h"
#include "commands/extension.h"
+#include "commands/matview.h"
#include "commands/repack.h"
#include "commands/sequence.h"
#include "commands/tablecmds.h"
@@ -1804,6 +1805,9 @@ RangeVarCallbackForDropRelation(const RangeVar *rel, Oid relOid, Oid oldRelOid,
state->actual_relkind = classform->relkind;
state->actual_relpersistence = classform->relpersistence;
+ if (classform->relkind == RELKIND_MATVIEW && classform->relisivm)
+ removeImmv(relOid);
+
/*
* Both RELKIND_RELATION and RELKIND_PARTITIONED_TABLE are OBJECT_TABLE,
* but RemoveRelations() can only pass one relkind for a given relation.
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 560659f9568..d97c35eeef0 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -370,6 +370,7 @@ WaitLSN "Waiting to read or update shared Wait-for-LSN state."
LogicalDecodingControl "Waiting to read or update logical decoding status information."
DataChecksumsWorker "Waiting for data checksums worker."
AioWorkerControl "Waiting to update AIO worker information."
+IvmControl "Waiting for read or update IMMV information.."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index be157a5fbe9..b8722eea7a4 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12693,4 +12693,14 @@
proname => 'hashoid8extended', prorettype => 'int8',
proargtypes => 'oid8 int8', prosrc => 'hashoid8extended' },
+# IVM
+{ oid => '786', descr => 'ivm trigger (before)',
+ proname => 'IVM_immediate_before', provolatile => 'v', prorettype => 'trigger',
+ proargtypes => '', prosrc => 'IVM_immediate_before' },
+{ oid => '787', descr => 'ivm trigger (after)',
+ proname => 'IVM_immediate_maintenance', provolatile => 'v', prorettype => 'trigger',
+ proargtypes => '', prosrc => 'IVM_immediate_maintenance' },
+{ oid => '788', descr => 'ivm filetring ',
+ proname => 'ivm_visible_in_prestate', provolatile => 's', prorettype => 'bool',
+ proargtypes => 'oid tid oid', prosrc => 'ivm_visible_in_prestate' },
]
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index f895c67c0c2..c286ebcd70e 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -16,6 +16,7 @@
#include "catalog/objectaddress.h"
#include "nodes/params.h"
+#include "nodes/pathnodes.h"
#include "parser/parse_node.h"
#include "tcop/dest.h"
#include "utils/queryenvironment.h"
@@ -25,6 +26,9 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
ParamListInfo params, QueryEnvironment *queryEnv,
QueryCompletion *qc);
+extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
+extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/commands/matview.h b/src/include/commands/matview.h
index 738c731c1a9..f1dbfc052c5 100644
--- a/src/include/commands/matview.h
+++ b/src/include/commands/matview.h
@@ -15,6 +15,7 @@
#define MATVIEW_H
#include "catalog/objectaddress.h"
+#include "fmgr.h"
#include "nodes/params.h"
#include "nodes/parsenodes.h"
#include "tcop/dest.h"
@@ -23,6 +24,8 @@
extern void SetMatViewPopulatedState(Relation relation, bool newstate);
+extern void SetMatViewIVMState(Relation relation, bool newstate);
+
extern ObjectAddress ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
QueryCompletion *qc);
extern ObjectAddress RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
@@ -33,4 +36,12 @@ extern DestReceiver *CreateTransientRelDestReceiver(Oid transientoid);
extern bool MatViewIncrementalMaintenanceIsEnabled(void);
+extern Datum IVM_immediate_before(PG_FUNCTION_ARGS);
+extern Datum IVM_immediate_maintenance(PG_FUNCTION_ARGS);
+extern Datum IVM_visible_in_prestate(PG_FUNCTION_ARGS);
+extern void AtAbort_IVM(SubTransactionId subtxid);
+extern void AtPreCommit_IVM(void);
+extern void removeImmv(Oid immv_oid);
+extern bool isIvmName(const char *s);
+
#endif /* MATVIEW_H */
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index d7eb648bd27..756c63baf97 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -89,6 +89,7 @@ PG_LWLOCK(54, WaitLSN)
PG_LWLOCK(55, LogicalDecodingControl)
PG_LWLOCK(56, DataChecksumsWorker)
PG_LWLOCK(57, AioWorkerControl)
+PG_LWLOCK(58, IvmControl)
/*
* There also exist several built-in LWLock tranches. As with the predefined
diff --git a/src/include/storage/subsystemlist.h b/src/include/storage/subsystemlist.h
index 9ad619080be..481e15d00a1 100644
--- a/src/include/storage/subsystemlist.h
+++ b/src/include/storage/subsystemlist.h
@@ -88,3 +88,6 @@ PG_SHMEM_SUBSYSTEM(DataChecksumsShmemCallbacks)
/* AIO subsystem. This delegates to the method-specific callbacks */
PG_SHMEM_SUBSYSTEM(AioShmemCallbacks)
+
+/* Incremental View Maintenance */
+PG_SHMEM_SUBSYSTEM(IvmShmemCallbacks)
--
2.43.0
--Multipart=_Fri__29_May_2026_23_14_17_+0900_Te0o73X2VqYK57Gd
Content-Type: text/x-diff;
name="v37-0005-Add-Incremental-View-Maintenance-support-to-psql.patch"
Content-Disposition: attachment;
filename="v37-0005-Add-Incremental-View-Maintenance-support-to-psql.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 40+ messages in thread
end of thread, other threads:[~2026-05-29 09:06 UTC | newest]
Thread overview: 40+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-01-12 03:03 Re: Time delayed LR (WAS Re: logical replication restrictions) Kyotaro Horiguchi <[email protected]>
2023-01-12 15:39 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-01-14 06:27 ` vignesh C <[email protected]>
2023-01-17 11:00 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-01-17 12:45 ` Hayato Kuroda (Fujitsu) <[email protected]>
2023-01-18 07:06 ` Peter Smith <[email protected]>
2023-01-19 01:41 ` Peter Smith <[email protected]>
2023-01-19 08:16 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-01-19 01:49 ` Peter Smith <[email protected]>
2023-01-19 06:35 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-01-19 10:54 ` vignesh C <[email protected]>
2023-01-19 12:59 ` Amit Kapila <[email protected]>
2023-01-19 13:23 ` vignesh C <[email protected]>
2023-01-20 19:07 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-01-19 13:17 ` Amit Kapila <[email protected]>
2023-01-20 18:57 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-01-20 06:55 ` Peter Smith <[email protected]>
2023-01-20 18:36 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-01-22 12:42 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-01-23 08:06 ` Peter Smith <[email protected]>
2023-01-23 10:44 ` Amit Kapila <[email protected]>
2023-01-23 22:15 ` Peter Smith <[email protected]>
2023-01-24 14:03 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-01-24 14:22 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-01-23 12:06 ` Amit Kapila <[email protected]>
2023-01-24 00:45 ` Kyotaro Horiguchi <[email protected]>
2023-01-23 23:32 ` Euler Taveira <[email protected]>
2023-01-24 05:18 ` Amit Kapila <[email protected]>
2023-01-24 14:57 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-02-06 11:56 ` Amit Kapila <[email protected]>
2023-02-06 13:21 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-01-19 07:12 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-01-20 03:46 ` shveta malik <[email protected]>
2023-01-20 07:38 ` Peter Smith <[email protected]>
2023-01-20 08:53 ` shveta malik <[email protected]>
2023-01-20 09:13 ` shveta malik <[email protected]>
2023-01-20 18:41 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-01-20 18:46 ` Takamichi Osumi (Fujitsu) <[email protected]>
2023-01-20 18:50 ` Takamichi Osumi (Fujitsu) <[email protected]>
2026-05-29 09:06 [PATCH v37 06/11] Add Incremental View Maintenance support Yugo Nagata <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox