public inbox for [email protected]
help / color / mirror / Atom feedFrom: [email protected] <[email protected]>
To: Amit Kapila <[email protected]>
Cc: [email protected] <[email protected]>
Cc: Peter Smith <[email protected]>
Cc: Dilip Kumar <[email protected]>
Cc: Masahiko Sawada <[email protected]>
Cc: [email protected] <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: RE: Perform streaming logical transactions by background workers and parallel apply
Date: Wed, 31 Aug 2022 09:55:45 +0000
Message-ID: <OS0PR01MB5716546863340CA19EC5F8C394789@OS0PR01MB5716.jpnprd01.prod.outlook.com> (raw)
In-Reply-To: <CAA4eK1Kom70AmcKjyKy1K77yCK1ipSUUGcaxXOHuRAqsRiZQmA@mail.gmail.com>
References: <OS3PR01MB62758DBE8FA12BA72A43AC819EB89@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<CAHut+Pu5ah+NhYMfb7KTbkYTxnuGY2j-7vT0G1MZZsS4SKiHew@mail.gmail.com>
<OS3PR01MB62755C6C9A75EB09F7218B589E839@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<OS3PR01MB627594D75E870BBB45E2E80A9E839@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<CAHut+PvN7fwtUE=bidzrsOUXSt+JpnkJztZ-Jn5t86moofaZ6g@mail.gmail.com>
<OS3PR01MB62758A6AAED27B3A848CEB7A9E8F9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<OS3PR01MB62752B29CEC2592E18236A9A9E909@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<CAFiTN-uq5Mp+o+_u0iDc8OQV-q7CWf5W=dPgw2ORwxsi6yjVfw@mail.gmail.com>
<CAA4eK1LB1s=B+krpn9mGbZq46pVei0FD7WjhJORq4bD2GU4fng@mail.gmail.com>
<CAFiTN-sK3bVppVAioPfxGjZqY9sHKVM8NooK9Wuex2OCTK25wg@mail.gmail.com>
<OS0PR01MB571651D89C744F84E20AAFE394979@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAA4eK1K0zkBJQtF1ZNhSvcwh_+pdCaUTf8+bBZR95yP92c0CuQ@mail.gmail.com>
<OS3PR01MB6275D64BE7726B0221B15F389E9F9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<CAHut+PutJBXuoy3DB5S7X9sZitfZp4qsXfzgkt5C5=CTNtX-Ug@mail.gmail.com>
<OS3PR01MB6275739E73E8BEC5D13FB6739E6B9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<OS3PR01MB6275BA4CBB62CBE8C094E53D9E6A9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<CAA4eK1+_oHZHoDooAR7QcYD2CeTUWNSwkqVcLWC2iQijAJC4Cg@mail.gmail.com>
<CAA4eK1++B2PFQiin2ZGMs_tszsQ=NJeKuj18-M4PEgckfEYqNA@mail.gmail.com>
<OS0PR01MB57166F15F4542FAE69A74EC094739@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAA4eK1JERguGiYoid9CMEOamZoD4jF_R8+mkNVS=mV-X=CXe=A@mail.gmail.com>
<OS0PR01MB5716DB4C67394D1676429D3594769@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAA4eK1+e8JsiC8uMZPU25xQRyxNvVS24M4=Zy-xD18jzX+vrmA@mail.gmail.com>
<CAA4eK1Kom70AmcKjyKy1K77yCK1ipSUUGcaxXOHuRAqsRiZQmA@mail.gmail.com>
On Tuesday, August 30, 2022 7:51 PM Amit Kapila <[email protected]> wrote:
>
> On Tue, Aug 30, 2022 at 12:12 PM Amit Kapila <[email protected]>
> wrote:
> >
> > Few other comments on v25-0001*
> > ============================
> >
>
> Some more comments on v25-0001*:
> =============================
> 1.
> +static void
> +apply_handle_stream_abort(StringInfo s)
> ...
> ...
> + else if (apply_action == TA_SEND_TO_PARALLEL_WORKER) { if (subxid ==
> + xid) parallel_apply_replorigin_reset();
> +
> + /* Send STREAM ABORT message to the apply parallel worker. */
> + parallel_apply_send_data(winfo, s->len, s->data);
> +
> + /*
> + * After sending the data to the apply parallel worker, wait for
> + * that worker to finish. This is necessary to maintain commit
> + * order which avoids failures due to transaction dependencies and
> + * deadlocks.
> + */
> + if (subxid == xid)
> + {
> + parallel_apply_wait_for_free(winfo);
> ...
> ...
>
> From this code, it appears that we are waiting for rollbacks to finish but not
> doing the same in the rollback to savepoint cases. Is there a reason for the
> same? I think we need to wait for rollbacks to avoid transaction dependency
> and deadlock issues. Consider the below case:
>
> Consider table t1 (c1 primary key, c2, c3) has a row (1, 2, 3) on both publisher and
> subscriber.
>
> Publisher
> Session-1
> ==========
> Begin;
> ...
> Delete from t1 where c1 = 1;
>
> Session-2
> Begin;
> ...
> insert into t1 values(1, 4, 5); --This will wait for Session-1's Delete to finish.
>
> Session-1
> Rollback;
>
> Session-2
> -- The wait will be finished and the insert will be successful.
> Commit;
>
> Now, assume both these transactions get streamed and if we didn't wait for
> rollback/rollback to savepoint, it is possible that the insert gets executed
> before and leads to a constraint violation. This won't happen in non-parallel
> mode, so we should wait for rollbacks to finish.
Agreed and changed.
> 2. I think we don't need to wait at Rollback Prepared/Commit Prepared
> because we wait for prepare to finish in *_stream_prepare function.
> That will ensure all the operations in that transaction have happened in the
> subscriber, so no concurrent transaction can create deadlock or transaction
> dependency issues. If so, I think it is better to explain this in the comments.
Added some comments about this.
> 3.
> +/* What action to take for the transaction. */ typedef enum
> {
> - LogicalRepMsgType command; /* 0 if invalid */
> - LogicalRepRelMapEntry *rel;
> + /* The action for non-streaming transactions. */
> + TA_APPLY_IN_LEADER_WORKER,
>
> - /* Remote node information */
> - int remote_attnum; /* -1 if invalid */
> - TransactionId remote_xid;
> - XLogRecPtr finish_lsn;
> - char *origin_name;
> -} ApplyErrorCallbackArg;
> + /* Actions for streaming transactions. */ TA_SERIALIZE_TO_FILE,
> +TA_APPLY_IN_PARALLEL_WORKER, TA_SEND_TO_PARALLEL_WORKER }
> +TransactionApplyAction;
>
> I think each action needs explanation atop this enum typedef.
Added.
> 4.
> @@ -1149,24 +1315,14 @@ static void
> apply_handle_stream_start(StringInfo s) { ...
> + else if (apply_action == TA_SERIALIZE_TO_FILE) {
> + /*
> + * For the first stream start, check if there is any free apply
> + * parallel worker we can use to process this transaction.
> + */
> + if (first_segment)
> + winfo = parallel_apply_start_worker(stream_xid);
>
> - /* open the spool file for this transaction */
> - stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
> + if (winfo)
> + {
> + /*
> + * If we have found a free worker, then we pass the data to that
> + * worker.
> + */
> + parallel_apply_send_data(winfo, s->len, s->data);
>
> - /* if this is not the first segment, open existing subxact file */
> - if (!first_segment)
> - subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
> + nchanges = 0;
>
> - pgstat_report_activity(STATE_RUNNING, NULL);
> + /* Cache the apply parallel worker for this transaction. */
> + stream_apply_worker = winfo; }
> ...
>
> This looks odd to me in the sense that even if the action is
> TA_SERIALIZE_TO_FILE, we still send the information to the parallel
> worker. Won't it be better if we call parallel_apply_start_worker()
> for first_segment before checking apply_action with
> get_transaction_apply_action(). That way we can avoid this special
> case handling.
Changed as suggested.
> 5.
> +/*
> + * Struct for sharing information between apply leader apply worker and apply
> + * parallel workers.
> + */
> +typedef struct ApplyParallelWorkerShared
> +{
> + slock_t mutex;
> +
> + bool in_use;
> +
> + /* Logical protocol version. */
> + uint32 proto_version;
> +
> + TransactionId stream_xid;
>
> Are we using stream_xid passed by the leader in parallel worker? If
> so, how? If not, then can we do without this?
No, it seems we don't need this. Removed.
> 6.
> +void
> +HandleParallelApplyMessages(void)
> {
> ...
> + /* OK to process messages. Reset the flag saying there are more to do. */
> + ParallelApplyMessagePending = false;
>
> I don't understand the meaning of the second part of the comment.
> Shouldn't we say: "Reset the flag saying there is nothing more to
> do."? I know you have copied from the other part of the code but there
> also I am not sure if it is correct.
I feel the comment here is not very helpful, so I removed this.
> 7.
> +static List *ApplyParallelWorkersFreeList = NIL;
> +static List *ApplyParallelWorkersList = NIL;
>
> Do we really need to maintain two different workers' lists? If so,
> what is the advantage? I think there won't be many parallel apply
> workers, so even if maintain one list and search it, there shouldn't
> be any performance impact. I feel maintaining two lists for this
> purpose is a bit complex and has more chances of bugs, so we should
> try to avoid it if possible.
Agreed, I removed the ApplyParallelWorkersList and reused
ApplyParallelWorkersList in other places.
Attach the new version patch set which addressed above comments
and comments from[1].
[1] https://www.postgresql.org/message-id/CAA4eK1%2Be8JsiC8uMZPU25xQRyxNvVS24M4%3DZy-xD18jzX%2BvrmA%40ma...
Best regards,
Hou zj
Attachments:
[application/octet-stream] v26-0005-Add-a-main_worker_pid-to-pg_stat_subscription.patch (7.6K, ../OS0PR01MB5716546863340CA19EC5F8C394789@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v26-0005-Add-a-main_worker_pid-to-pg_stat_subscription.patch)
download | inline diff:
From 7a00e56b6fb71d1cb5874e162775f7353aa52873 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <[email protected]>
Date: Thu, 11 Aug 2022 11:49:08 +0800
Subject: [PATCH v26 5/5] Add a main_worker_pid to pg_stat_subscription
main_worker_pid is Process ID of the leader apply worker, if this process is a
apply parallel worker. NULL if this process is a leader apply worker or a
synchronization worker.
The new column can make it easier to distinguish leader apply worker and apply
parallel worker which is also similar to the 'leader_pid' column in
pg_stat_activity.
---
doc/src/sgml/monitoring.sgml | 21 +++++++++++++++++----
src/backend/catalog/system_views.sql | 1 +
src/backend/replication/logical/launcher.c | 28 +++++++++++++++++-----------
src/include/catalog/pg_proc.dat | 6 +++---
src/test/regress/expected/rules.out | 3 ++-
5 files changed, 40 insertions(+), 19 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1d9509a..2c2e961 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3180,11 +3180,22 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<row>
<entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>apply_leader_pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of the leader apply worker, if this process is an apply
+ parallel worker. NULL if this process is a leader apply worker or a
+ synchronization worker.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
<structfield>relid</structfield> <type>oid</type>
</para>
<para>
OID of the relation that the worker is synchronizing; null for the
- main apply worker
+ main apply worker and the apply parallel worker
</para></entry>
</row>
@@ -3194,7 +3205,7 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
</para>
<para>
Last write-ahead log location received, the initial value of
- this field being 0
+ this field being 0; null for the apply parallel worker
</para></entry>
</row>
@@ -3203,7 +3214,8 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<structfield>last_msg_send_time</structfield> <type>timestamp with time zone</type>
</para>
<para>
- Send time of last message received from origin WAL sender
+ Send time of last message received from origin WAL sender; null for the
+ apply parallel worker
</para></entry>
</row>
@@ -3212,7 +3224,8 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<structfield>last_msg_receipt_time</structfield> <type>timestamp with time zone</type>
</para>
<para>
- Receipt time of last message received from origin WAL sender
+ Receipt time of last message received from origin WAL sender; null for
+ the apply parallel worker
</para></entry>
</row>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 652d76f..d35748b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -947,6 +947,7 @@ CREATE VIEW pg_stat_subscription AS
su.oid AS subid,
su.subname,
st.pid,
+ st.apply_leader_pid,
st.relid,
st.received_lsn,
st.last_msg_send_time,
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 7a46cc2..99f491c 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -1045,7 +1045,7 @@ IsLogicalLauncher(void)
Datum
pg_stat_get_subscription(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_SUBSCRIPTION_COLS 8
+#define PG_STAT_GET_SUBSCRIPTION_COLS 9
Oid subid = PG_ARGISNULL(0) ? InvalidOid : PG_GETARG_OID(0);
int i;
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
@@ -1079,26 +1079,32 @@ pg_stat_get_subscription(PG_FUNCTION_ARGS)
else
nulls[1] = true;
values[2] = Int32GetDatum(worker_pid);
- if (XLogRecPtrIsInvalid(worker.last_lsn))
+
+ if (worker.apply_leader_pid == 0)
nulls[3] = true;
else
- values[3] = LSNGetDatum(worker.last_lsn);
- if (worker.last_send_time == 0)
+ values[3] = Int32GetDatum(worker.apply_leader_pid);
+
+ if (XLogRecPtrIsInvalid(worker.last_lsn))
nulls[4] = true;
else
- values[4] = TimestampTzGetDatum(worker.last_send_time);
- if (worker.last_recv_time == 0)
+ values[4] = LSNGetDatum(worker.last_lsn);
+ if (worker.last_send_time == 0)
nulls[5] = true;
else
- values[5] = TimestampTzGetDatum(worker.last_recv_time);
- if (XLogRecPtrIsInvalid(worker.reply_lsn))
+ values[5] = TimestampTzGetDatum(worker.last_send_time);
+ if (worker.last_recv_time == 0)
nulls[6] = true;
else
- values[6] = LSNGetDatum(worker.reply_lsn);
- if (worker.reply_time == 0)
+ values[6] = TimestampTzGetDatum(worker.last_recv_time);
+ if (XLogRecPtrIsInvalid(worker.reply_lsn))
nulls[7] = true;
else
- values[7] = TimestampTzGetDatum(worker.reply_time);
+ values[7] = LSNGetDatum(worker.reply_lsn);
+ if (worker.reply_time == 0)
+ nulls[8] = true;
+ else
+ values[8] = TimestampTzGetDatum(worker.reply_time);
tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
values, nulls);
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index be47583..b68843d 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5393,9 +5393,9 @@
proname => 'pg_stat_get_subscription', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => 'oid',
- proallargtypes => '{oid,oid,oid,int4,pg_lsn,timestamptz,timestamptz,pg_lsn,timestamptz}',
- proargmodes => '{i,o,o,o,o,o,o,o,o}',
- proargnames => '{subid,subid,relid,pid,received_lsn,last_msg_send_time,last_msg_receipt_time,latest_end_lsn,latest_end_time}',
+ proallargtypes => '{oid,oid,oid,int4,int4,pg_lsn,timestamptz,timestamptz,pg_lsn,timestamptz}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{subid,subid,relid,pid,apply_leader_pid,received_lsn,last_msg_send_time,last_msg_receipt_time,latest_end_lsn,latest_end_time}',
prosrc => 'pg_stat_get_subscription' },
{ oid => '2026', descr => 'statistics: current backend PID',
proname => 'pg_backend_pid', provolatile => 's', proparallel => 'r',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 7ec3d26..007e710 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2091,6 +2091,7 @@ pg_stat_ssl| SELECT s.pid,
pg_stat_subscription| SELECT su.oid AS subid,
su.subname,
st.pid,
+ st.apply_leader_pid,
st.relid,
st.received_lsn,
st.last_msg_send_time,
@@ -2098,7 +2099,7 @@ pg_stat_subscription| SELECT su.oid AS subid,
st.latest_end_lsn,
st.latest_end_time
FROM (pg_subscription su
- LEFT JOIN pg_stat_get_subscription(NULL::oid) st(subid, relid, pid, received_lsn, last_msg_send_time, last_msg_receipt_time, latest_end_lsn, latest_end_time) ON ((st.subid = su.oid)));
+ LEFT JOIN pg_stat_get_subscription(NULL::oid) st(subid, relid, pid, apply_leader_pid, received_lsn, last_msg_send_time, last_msg_receipt_time, latest_end_lsn, latest_end_time) ON ((st.subid = su.oid)));
pg_stat_subscription_stats| SELECT ss.subid,
s.subname,
ss.apply_error_count,
--
2.7.2.windows.1
[application/octet-stream] v26-0001-Perform-streaming-logical-transactions-by-parall.patch (129.2K, ../OS0PR01MB5716546863340CA19EC5F8C394789@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v26-0001-Perform-streaming-logical-transactions-by-parall.patch)
download | inline diff:
From ed5ea1420d22d635407bb6e6d26daef8462dd524 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <[email protected]>
Date: Wed, 20 Apr 2022 16:45:07 +0800
Subject: [PATCH v26 1/5] Perform streaming logical transactions by parallel
workers
Currently, for large transactions, the publisher sends the data in multiple
streams (changes divided into chunks depending upon logical_decoding_work_mem),
and then on the subscriber-side, the apply worker writes the changes into
temporary files and once it receives the commit, it reads from the file and
applies the entire transaction. To improve the performance of such
transactions, we can instead allow them to be applied via parallel workers.
In this approach, we assign a new parallel apply worker (if available) as
soon as the xact's first stream is received and the leader apply worker will
send changes to this new worker via shared memory. The parallel apply worker
will directly apply the change instead of writing it to temporary files. We
keep this worker assigned till the transaction commit is received and also wait
for the worker to finish at commit. This preserves commit ordering and avoids
writing to and reading from file in most cases. We still need to spill if there
is no worker available.
This patch also extends the SUBSCRIPTION 'streaming' parameter so that the user
can control whether to apply the streaming transaction in an parallel apply
worker or spill the change to disk. The user can set the streaming parameter to
'on/off', 'parallel'. The parameter value 'parallel' means the streaming will
be applied via an parallel apply worker, if available. The parameter value
'on' means the streaming transaction will be spilled to disk. The default value
is 'off' (same as current behaviour).
In addition, the patch extends the logical replication STREAM_ABORT message so
that abort_time and abort_lsn can also be sent which can be used to update the
replication origin in parallel apply worker when the streaming transaction is
aborted.
---
doc/src/sgml/catalogs.sgml | 11 +-
doc/src/sgml/config.sgml | 28 +-
doc/src/sgml/logical-replication.sgml | 13 +-
doc/src/sgml/protocol.sgml | 29 +-
doc/src/sgml/ref/create_subscription.sgml | 24 +-
src/backend/access/transam/xact.c | 13 +
src/backend/commands/subscriptioncmds.c | 66 +-
src/backend/libpq/pqmq.c | 18 +-
src/backend/postmaster/bgworker.c | 3 +
src/backend/replication/logical/Makefile | 1 +
.../replication/logical/applyparallelworker.c | 1089 ++++++++++++++++++++
src/backend/replication/logical/decode.c | 10 +-
src/backend/replication/logical/launcher.c | 166 ++-
src/backend/replication/logical/proto.c | 41 +-
src/backend/replication/logical/reorderbuffer.c | 10 +-
src/backend/replication/logical/tablesync.c | 6 +-
src/backend/replication/logical/worker.c | 830 +++++++++++----
src/backend/replication/pgoutput/pgoutput.c | 6 +-
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/tcop/postgres.c | 3 +
src/backend/utils/activity/wait_event.c | 3 +
src/backend/utils/misc/guc.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/bin/pg_dump/pg_dump.c | 6 +-
src/include/catalog/pg_subscription.h | 21 +-
src/include/nodes/pg_list.h | 11 +
src/include/replication/logicallauncher.h | 1 +
src/include/replication/logicalproto.h | 27 +-
src/include/replication/logicalworker.h | 6 +
src/include/replication/reorderbuffer.h | 7 +-
src/include/replication/worker_internal.h | 124 ++-
src/include/storage/procsignal.h | 1 +
src/include/utils/wait_event.h | 1 +
src/test/regress/expected/subscription.out | 12 +-
src/test/regress/sql/subscription.sql | 6 +-
src/tools/pgindent/typedefs.list | 4 +
36 files changed, 2362 insertions(+), 252 deletions(-)
create mode 100644 src/backend/replication/logical/applyparallelworker.c
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 00f833d..e5e8b4d 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7892,11 +7892,16 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>substream</structfield> <type>bool</type>
+ <structfield>substream</structfield> <type>char</type>
</para>
<para>
- If true, the subscription will allow streaming of in-progress
- transactions
+ Controls how to handle the streaming of in-progress transactions:
+ <literal>f</literal> = disallow streaming of in-progress transactions,
+ <literal>t</literal> = spill the changes of in-progress transactions to
+ disk and apply at once after the transaction is committed on the
+ publisher,
+ <literal>p</literal> = apply changes directly using an parallel apply
+ worker if available (same as 't' if no worker is available)
</para></entry>
</row>
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index a5cd4e4..a051490 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4940,7 +4940,8 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
<listitem>
<para>
Specifies maximum number of logical replication workers. This includes
- both apply workers and table synchronization workers.
+ apply leader workers, parallel apply workers, and table synchronization
+ workers.
</para>
<para>
Logical replication workers are taken from the pool defined by
@@ -4980,6 +4981,31 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-max-parallel-apply-workers-per-subscription" xreflabel="max_parallel_apply_workers_per_subscription">
+ <term><varname>max_parallel_apply_workers_per_subscription</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>max_parallel_apply_workers_per_subscription</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Maximum number of parallel apply workers per subscription. This
+ parameter controls the amount of parallelism for streaming of
+ in-progress transactions with subscription parameter
+ <literal>streaming = parallel</literal>.
+ </para>
+ <para>
+ The parallel apply workers are taken from the pool defined by
+ <varname>max_logical_replication_workers</varname>.
+ </para>
+ <para>
+ The default value is 2. This parameter can only be set in the
+ <filename>postgresql.conf</filename> file or on the server command
+ line.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index bdf1e7b..a24ca10 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1153,6 +1153,16 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
might not violate any constraint. This can easily make the subscriber
inconsistent.
</para>
+
+ <para>
+ When the streaming mode is <literal>parallel</literal>, the finish LSN of
+ failed transactions may not be logged. In that case, it may be necessary to
+ change the streaming mode to <literal>on</literal> or <literal>off</literal> and
+ cause the same conflicts again so the finish LSN of the failed transaction will
+ be written to the server log. For the usage of finish LSN, please refer to <link
+ linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ...
+ SKIP</command></link>.
+ </para>
</sect1>
<sect1 id="logical-replication-restrictions">
@@ -1334,7 +1344,8 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
subscription. A disabled subscription or a crashed subscription will have
zero rows in this view. If the initial data synchronization of any
table is in progress, there will be additional workers for the tables
- being synchronized.
+ being synchronized. Moreover, if the streaming transaction is applied in
+ parallel, there will be additional workers.
</para>
</sect1>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 87870c5..d107dd2 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3096,7 +3096,7 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<listitem>
<para>
Protocol version. Currently versions <literal>1</literal>, <literal>2</literal>,
- and <literal>3</literal> are supported.
+ <literal>3</literal>, and <literal>4</literal> are supported.
</para>
<para>
Version <literal>2</literal> is supported only for server version 14
@@ -3106,6 +3106,11 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
Version <literal>3</literal> is supported only for server version 15
and above, and it allows streaming of two-phase commits.
</para>
+ <para>
+ Version <literal>4</literal> is supported only for server version 16
+ and above, and it allows applying streams of large in-progress
+ transactions in parallel.
+ </para>
</listitem>
</varlistentry>
@@ -6876,6 +6881,28 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term>Int64 (XLogRecPtr)</term>
+ <listitem>
+ <para>
+ The LSN of the abort. This field is available since protocol version
+ 4.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>Int64 (TimestampTz)</term>
+ <listitem>
+ <para>
+ Abort timestamp of the transaction. The value is in number
+ of microseconds since PostgreSQL epoch (2000-01-01). This field is
+ available since protocol version 4.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 7390c71..cbcd02b 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -217,13 +217,29 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</varlistentry>
<varlistentry>
- <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <term><literal>streaming</literal> (<type>enum</type>)</term>
<listitem>
<para>
Specifies whether to enable streaming of in-progress transactions
- for this subscription. By default, all transactions
- are fully decoded on the publisher and only then sent to the
- subscriber as a whole.
+ for this subscription. The default value is <literal>off</literal>,
+ meaning all transactions are fully decoded on the publisher and only
+ then sent to the subscriber as a whole.
+ </para>
+
+ <para>
+ If set to <literal>on</literal>, the incoming changes are written to
+ temporary files and then applied only after the transaction is
+ committed on the publisher and received by the subscriber.
+ </para>
+
+ <para>
+ If set to <literal>parallel</literal>, incoming changes are directly
+ applied via one of the parallel apply workers, if available. If no
+ parallel worker is free to handle streaming transactions then the
+ changes are written to temporary files and applied after the
+ transaction is committed. Note that if an error happens when
+ applying changes in a parallel worker, the finish LSN of the
+ remote transaction might not be reported in the server log.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 50f092d..5a76d9c 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -1711,6 +1711,7 @@ RecordTransactionAbort(bool isSubXact)
int nchildren;
TransactionId *children;
TimestampTz xact_time;
+ bool replorigin;
/*
* If we haven't been assigned an XID, nobody will care whether we aborted
@@ -1741,6 +1742,13 @@ RecordTransactionAbort(bool isSubXact)
elog(PANIC, "cannot abort transaction %u, it was already committed",
xid);
+ /*
+ * Are we using the replication origins feature? Or, in other words,
+ * are we replaying remote actions?
+ */
+ replorigin = (replorigin_session_origin != InvalidRepOriginId &&
+ replorigin_session_origin != DoNotReplicateId);
+
/* Fetch the data we need for the abort record */
nrels = smgrGetPendingDeletes(false, &rels);
nchildren = xactGetCommittedChildren(&children);
@@ -1765,6 +1773,11 @@ RecordTransactionAbort(bool isSubXact)
MyXactFlags, InvalidTransactionId,
NULL);
+ if (replorigin)
+ /* Move LSNs forward for this replication origin */
+ replorigin_session_advance(replorigin_session_origin_lsn,
+ XactLastRecEnd);
+
/*
* Report the latest async abort LSN, so that the WAL writer knows to
* flush this abort. There's nothing to be gained by delaying this, since
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 670b219..3dc03dc 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -84,7 +84,7 @@ typedef struct SubOpts
bool copy_data;
bool refresh;
bool binary;
- bool streaming;
+ char streaming;
bool twophase;
bool disableonerr;
char *origin;
@@ -98,6 +98,62 @@ static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname,
/*
+ * Extract the streaming mode value from a DefElem. This is like
+ * defGetBoolean() but also accepts the special value of "parallel".
+ */
+static char
+defGetStreamingMode(DefElem *def)
+{
+ /*
+ * If no parameter value given, assume "true" is meant.
+ */
+ if (def->arg == NULL)
+ return SUBSTREAM_ON;
+
+ /*
+ * Allow 0, 1, "false", "true", "off", "on" or "parallel".
+ */
+ switch (nodeTag(def->arg))
+ {
+ case T_Integer:
+ switch (intVal(def->arg))
+ {
+ case 0:
+ return SUBSTREAM_OFF;
+ case 1:
+ return SUBSTREAM_ON;
+ default:
+ /* otherwise, error out below */
+ break;
+ }
+ break;
+ default:
+ {
+ char *sval = defGetString(def);
+
+ /*
+ * The set of strings accepted here should match up with the
+ * grammar's opt_boolean_or_string production.
+ */
+ if (pg_strcasecmp(sval, "false") == 0 ||
+ pg_strcasecmp(sval, "off") == 0)
+ return SUBSTREAM_OFF;
+ if (pg_strcasecmp(sval, "true") == 0 ||
+ pg_strcasecmp(sval, "on") == 0)
+ return SUBSTREAM_ON;
+ if (pg_strcasecmp(sval, "parallel") == 0)
+ return SUBSTREAM_PARALLEL;
+ }
+ break;
+ }
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s requires a Boolean value or \"parallel\"",
+ def->defname)));
+ return SUBSTREAM_OFF; /* keep compiler quiet */
+}
+
+/*
* Common option parsing function for CREATE and ALTER SUBSCRIPTION commands.
*
* Since not all options can be specified in both commands, this function
@@ -134,7 +190,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
if (IsSet(supported_opts, SUBOPT_BINARY))
opts->binary = false;
if (IsSet(supported_opts, SUBOPT_STREAMING))
- opts->streaming = false;
+ opts->streaming = SUBSTREAM_OFF;
if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT))
opts->twophase = false;
if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
@@ -237,7 +293,7 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
errorConflictingDefElem(defel, pstate);
opts->specified_opts |= SUBOPT_STREAMING;
- opts->streaming = defGetBoolean(defel);
+ opts->streaming = defGetStreamingMode(defel);
}
else if (strcmp(defel->defname, "two_phase") == 0)
{
@@ -627,7 +683,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled);
values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary);
- values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming);
+ values[Anum_pg_subscription_substream - 1] = CharGetDatum(opts.streaming);
values[Anum_pg_subscription_subtwophasestate - 1] =
CharGetDatum(opts.twophase ?
LOGICALREP_TWOPHASE_STATE_PENDING :
@@ -1089,7 +1145,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
if (IsSet(opts.specified_opts, SUBOPT_STREAMING))
{
values[Anum_pg_subscription_substream - 1] =
- BoolGetDatum(opts.streaming);
+ CharGetDatum(opts.streaming);
replaces[Anum_pg_subscription_substream - 1] = true;
}
diff --git a/src/backend/libpq/pqmq.c b/src/backend/libpq/pqmq.c
index 4d0415e..6d5394e 100644
--- a/src/backend/libpq/pqmq.c
+++ b/src/backend/libpq/pqmq.c
@@ -13,11 +13,13 @@
#include "postgres.h"
+#include "access/parallel.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "libpq/pqmq.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "replication/logicalworker.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
@@ -162,9 +164,19 @@ mq_putmessage(char msgtype, const char *s, size_t len)
result = shm_mq_sendv(pq_mq_handle, iov, 2, true, true);
if (pq_mq_parallel_leader_pid != 0)
- SendProcSignal(pq_mq_parallel_leader_pid,
- PROCSIG_PARALLEL_MESSAGE,
- pq_mq_parallel_leader_backend_id);
+ {
+ if (IsParallelWorker())
+ SendProcSignal(pq_mq_parallel_leader_pid,
+ PROCSIG_PARALLEL_MESSAGE,
+ pq_mq_parallel_leader_backend_id);
+ else
+ {
+ Assert(IsLogicalParallelApplyWorker());
+ SendProcSignal(pq_mq_parallel_leader_pid,
+ PROCSIG_PARALLEL_APPLY_MESSAGE,
+ pq_mq_parallel_leader_backend_id);
+ }
+ }
if (result != SHM_MQ_WOULD_BLOCK)
break;
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index 40601ae..ee481bb 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -128,6 +128,9 @@ static const struct
},
{
"ApplyWorkerMain", ApplyWorkerMain
+ },
+ {
+ "ParallelApplyWorkerMain", ParallelApplyWorkerMain
}
};
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index c4e2fde..2dc25e3 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -15,6 +15,7 @@ include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I$(srcdir) $(CPPFLAGS)
OBJS = \
+ applyparallelworker.o \
decode.o \
launcher.o \
logical.o \
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
new file mode 100644
index 0000000..f592c2b
--- /dev/null
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -0,0 +1,1089 @@
+/*-------------------------------------------------------------------------
+ * applyparallelworker.c
+ * Support routines for applying xact by parallel apply worker
+ *
+ * Copyright (c) 2016-2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/applyparallelworker.c
+ *
+ * This file contains routines that are intended to support setting up, using,
+ * and tearing down a ParallelApplyWorkerInfo.
+ *
+ * Refer to the comments in file header of logical/worker.c to see more
+ * information about parallel apply worker.
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "libpq/pqformat.h"
+#include "libpq/pqmq.h"
+#include "mb/pg_wchar.h"
+#include "pgstat.h"
+#include "postmaster/interrupt.h"
+#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
+#include "replication/origin.h"
+#include "replication/walreceiver.h"
+#include "replication/worker_internal.h"
+#include "storage/ipc.h"
+#include "storage/procarray.h"
+#include "tcop/tcopprot.h"
+#include "utils/inval.h"
+#include "utils/memutils.h"
+#include "utils/resowner.h"
+#include "utils/syscache.h"
+
+#define PG_LOGICAL_APPLY_SHM_MAGIC 0x787ca067
+
+/*
+ * DSM keys for parallel apply worker. Unlike other parallel execution code,
+ * since we don't need to worry about DSM keys conflicting with plan_node_id we
+ * can use small integers.
+ */
+#define PARALLEL_APPLY_KEY_SHARED 1
+#define PARALLEL_APPLY_KEY_MQ 2
+#define PARALLEL_APPLY_KEY_ERROR_QUEUE 3
+
+/* Queue size of DSM, 16 MB for now. */
+#define DSM_QUEUE_SIZE (16 * 1024 * 1024)
+
+/*
+ * Error queue size of DSM. It is desirable to make it large enough that a
+ * typical ErrorResponse can be sent without blocking. That way, a worker that
+ * errors out can write the whole message into the queue and terminate without
+ * waiting for the user backend.
+ */
+#define DSM_ERROR_QUEUE_SIZE 16384
+
+/*
+ * There are three fields in each message received by apply worker: start_lsn,
+ * end_lsn and send_time. Because we have updated these statistics in apply
+ * worker, we could ignore these fields in parallel apply worker (see function
+ * LogicalRepApplyLoop).
+ */
+#define SIZE_STATS_MESSAGE (2 * sizeof(XLogRecPtr) + sizeof(TimestampTz))
+
+/*
+ * Entry for a hash table we use to map from xid to the parallel apply worker
+ * state.
+ */
+typedef struct ParallelApplyWorkerEntry
+{
+ TransactionId xid; /* Hash key -- must be first */
+ ParallelApplyWorkerInfo *winfo;
+} ParallelApplyWorkerEntry;
+
+/* Parallel apply workers hash table (initialized on first use). */
+static HTAB *ParallelApplyWorkersHash = NULL;
+
+/*
+ * List that stores the information of parallel apply workers that were
+ * started. Newly added worker information will be removed from the list at the
+ * end of the transaction when there are enough workers in the pool. Besides,
+ * exited workers will be removed from the list after being detected.
+*/
+static List *ParallelApplyWorkersList = NIL;
+
+/*
+ * Information shared between leader apply worker and parallel apply worker.
+ */
+volatile ParallelApplyWorkerShared *MyParallelShared = NULL;
+
+/*
+ * Is there a message pending in parallel apply worker which we need to
+ * receive?
+ */
+volatile bool ParallelApplyMessagePending = false;
+
+/*
+ * Cache a pointer to the parallel apply worker which is responsible for
+ * applying the current streaming transactions. It is used to save the
+ * cost of searching the hash table when applying the changes between
+ * STREAM_START and STREAM_STOP.
+ */
+ParallelApplyWorkerInfo *stream_apply_worker = NULL;
+
+List *subxactlist = NIL;
+
+static bool parallel_apply_can_start(TransactionId xid);
+static bool parallel_apply_setup_dsm(ParallelApplyWorkerInfo *winfo);
+static ParallelApplyWorkerInfo *parallel_apply_setup_worker(void);
+
+/*
+ * Check if starting a new parallel apply worker is allowed.
+ */
+static bool
+parallel_apply_can_start(TransactionId xid)
+{
+ if (!TransactionIdIsValid(xid))
+ return false;
+
+ /*
+ * Don't start a new parallel worker if not in streaming parallel mode.
+ */
+ if (MySubscription->stream != SUBSTREAM_PARALLEL)
+ return false;
+
+ /*
+ * Don't start a new parallel worker if user has set skiplsn as it's
+ * possible that user want to skip the streaming transaction. For
+ * streaming transaction, we need to spill the transaction to disk so that
+ * we can get the last LSN of the transaction to judge whether to skip
+ * before starting to apply the change.
+ */
+ if (!XLogRecPtrIsInvalid(MySubscription->skiplsn))
+ return false;
+
+ /*
+ * For streaming transactions that are being applied using parallel apply
+ * worker, we cannot decide whether to apply the change for a relation that
+ * is not in the READY state (see should_apply_changes_for_rel) as we won't
+ * know remote_final_lsn by that time. So, we don't start the new parallel
+ * apply worker in this case.
+ */
+ if (!AllTablesyncsReady())
+ return false;
+
+ /*
+ * Do not allow parallel apply worker to be started in the parallel apply
+ * worker.
+ */
+ if (am_parallel_apply_worker())
+ return false;
+
+ return true;
+}
+
+/*
+ * Start a parallel apply worker information that will be used for the
+ * specified xid.
+ *
+ * If an parallel apply worker is not in use then re-use it, otherwise start a
+ * fresh one. Cache the worker information in ParallelApplyWorkersHash keyed by
+ * the specified xid.
+ */
+void
+parallel_apply_start_worker(TransactionId xid)
+{
+ bool found;
+ ListCell *lc;
+ ParallelApplyWorkerInfo *winfo = NULL;
+ ParallelApplyWorkerEntry *entry = NULL;
+
+ if (!parallel_apply_can_start(xid))
+ return;
+
+ /* First time through, initialize apply workers hashtable. */
+ if (ParallelApplyWorkersHash == NULL)
+ {
+ HASHCTL ctl;
+
+ MemSet(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(TransactionId);
+ ctl.entrysize = sizeof(ParallelApplyWorkerEntry);
+ ctl.hcxt = ApplyContext;
+
+ ParallelApplyWorkersHash = hash_create("logical apply workers hash",
+ 16, &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ }
+
+ /* Try to get a free parallel apply worker. */
+ foreach(lc, ParallelApplyWorkersList)
+ {
+ ParallelApplyWorkerInfo *tmp_winfo;
+
+ tmp_winfo = (ParallelApplyWorkerInfo *) lfirst(lc);
+
+ if (tmp_winfo->error_mq_handle == NULL)
+ {
+ /*
+ * Release the worker information and try next one if the parallel
+ * apply worker exited cleanly.
+ */
+ ParallelApplyWorkersList = foreach_delete_current(ParallelApplyWorkersList, lc);
+ shm_mq_detach(tmp_winfo->mq_handle);
+ dsm_detach(tmp_winfo->dsm_seg);
+ pfree(tmp_winfo);
+
+ continue;
+ }
+
+ if (!tmp_winfo->in_remote_transaction)
+ {
+ winfo = tmp_winfo;
+ break;
+ }
+ }
+
+ /* Try to start a new parallel apply worker. */
+ if (winfo == NULL)
+ winfo = parallel_apply_setup_worker();
+
+ /* Failed to start a new parallel apply worker. */
+ if (winfo == NULL)
+ return;
+
+ /* Create entry for requested transaction. */
+ entry = hash_search(ParallelApplyWorkersHash, &xid, HASH_ENTER, &found);
+ if (found)
+ elog(ERROR, "hash table corrupted");
+
+ winfo->in_remote_transaction = true;
+ winfo->shared->in_use = true;
+ entry->winfo = winfo;
+ entry->xid = xid;
+}
+
+/*
+ * Find the previously assigned worker for the given transaction, if any.
+ */
+ParallelApplyWorkerInfo *
+parallel_apply_find_worker(TransactionId xid)
+{
+ bool found;
+ ParallelApplyWorkerEntry *entry = NULL;
+
+ if (!TransactionIdIsValid(xid))
+ return NULL;
+
+ if (ParallelApplyWorkersHash == NULL)
+ return NULL;
+
+ /* Return the cached parallel apply worker if valid. */
+ if (stream_apply_worker != NULL)
+ return stream_apply_worker;
+
+ /*
+ * Find entry for requested transaction.
+ */
+ entry = hash_search(ParallelApplyWorkersHash, &xid, HASH_FIND, &found);
+ if (found)
+ {
+ /* If any workers (or the postmaster) have died, we have failed. */
+ if (entry->winfo->error_mq_handle == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("lost connection to parallel apply worker")));
+
+ entry->winfo->shared->in_use = true;
+
+ return entry->winfo;
+ }
+
+ return NULL;
+}
+
+/*
+ * Remove the parallel apply worker entry from the hash table. And stop the
+ * worker if there are enough workers in the pool.
+ *
+ * Note that this function should only be called at the end of the transaction.
+ */
+void
+parallel_apply_free_worker(ParallelApplyWorkerInfo *winfo, TransactionId xid)
+{
+ int napplyworkers;
+
+ Assert(!winfo->shared->in_use);
+
+ hash_search(ParallelApplyWorkersHash, &xid, HASH_REMOVE, NULL);
+
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+ napplyworkers = logicalrep_parallel_apply_worker_count(MyLogicalRepWorker->subid);
+ LWLockRelease(LogicalRepWorkerLock);
+
+ if (napplyworkers > (max_parallel_apply_workers_per_subscription / 2))
+ {
+ logicalrep_worker_stop_by_slot(winfo->shared->logicalrep_worker_slot_no,
+ winfo->shared->logicalrep_worker_generation);
+
+ ParallelApplyWorkersList = list_delete_ptr(ParallelApplyWorkersList, winfo);
+
+ shm_mq_detach(winfo->mq_handle);
+ shm_mq_detach(winfo->error_mq_handle);
+ dsm_detach(winfo->dsm_seg);
+ pfree(winfo);
+ }
+ else
+ winfo->in_remote_transaction = false;
+}
+
+/* Parallel apply worker main loop. */
+static void
+LogicalParallelApplyLoop(shm_mq_handle *mqh, volatile ParallelApplyWorkerShared *shared)
+{
+ shm_mq_result shmq_res;
+ PGPROC *registrant;
+ ErrorContextCallback errcallback;
+
+ registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid);
+ SetLatch(®istrant->procLatch);
+
+ /*
+ * Init the ApplyMessageContext which we clean up after each replication
+ * protocol message.
+ */
+ ApplyMessageContext = AllocSetContextCreate(ApplyContext,
+ "ApplyMessageContext",
+ ALLOCSET_DEFAULT_SIZES);
+
+ /*
+ * Push apply error context callback. Fields will be filled while applying
+ * a change.
+ */
+ errcallback.callback = apply_error_callback;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ for (;;)
+ {
+ void *data;
+ Size len;
+ int c;
+ StringInfoData s;
+ MemoryContext oldctx;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* Ensure we are reading the data into our memory context. */
+ oldctx = MemoryContextSwitchTo(ApplyMessageContext);
+
+ shmq_res = shm_mq_receive(mqh, &len, &data, false);
+
+ if (shmq_res != SHM_MQ_SUCCESS)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("lost connection to the leader apply worker")));
+
+ if (len == 0)
+ break;
+
+ s.cursor = 0;
+ s.maxlen = -1;
+ s.data = (char *) data;
+ s.len = len;
+
+ /*
+ * The first byte of message for additional communication between
+ * leader apply worker and parallel apply workers can only be 'w'.
+ */
+ c = pq_getmsgbyte(&s);
+ if (c != 'w')
+ elog(ERROR, "[Parallel Apply Worker #%u] unexpected message \"%c\"",
+ shared->worker_id, c);
+
+ /*
+ * Ignore statistics fields that have been updated by the leader apply
+ * worker.
+ */
+ s.cursor += SIZE_STATS_MESSAGE;
+
+ apply_dispatch(&s);
+
+ MemoryContextSwitchTo(oldctx);
+ MemoryContextReset(ApplyMessageContext);
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+
+ /* Pop the error context stack. */
+ error_context_stack = errcallback.previous;
+
+ elog(DEBUG1, "[Parallel Apply Worker #%u] exiting", shared->worker_id);
+
+ /* Signal main process that we are done. */
+ SetLatch(®istrant->procLatch);
+}
+
+/*
+ * Make sure the leader apply worker tries to read from our error queue one more
+ * time. This guards against the case where we exit uncleanly without sending
+ * an ErrorResponse, for example because some code calls proc_exit directly.
+ */
+static void
+parallel_apply_shutdown(int code, Datum arg)
+{
+ SendProcSignal(MyLogicalRepWorker->apply_leader_pid,
+ PROCSIG_PARALLEL_APPLY_MESSAGE,
+ InvalidBackendId);
+
+ dsm_detach((dsm_segment *) DatumGetPointer(arg));
+}
+
+/*
+ * Parallel apply worker entry point.
+ */
+void
+ParallelApplyWorkerMain(Datum main_arg)
+{
+ volatile ParallelApplyWorkerShared *shared;
+ dsm_handle handle;
+ dsm_segment *seg;
+ shm_toc *toc;
+ shm_mq *mq;
+ shm_mq_handle *mqh;
+ shm_mq_handle *error_mqh;
+ int worker_slot = DatumGetInt32(main_arg);
+ char originname[NAMEDATALEN];
+
+ /* Setup signal handling. */
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /*
+ * Attach to the dynamic shared memory segment for the parallel apply, and
+ * find its table of contents.
+ *
+ * Like parallel query, we don't need resource owner by this time. See
+ * ParallelWorkerMain.
+ */
+ memcpy(&handle, MyBgworkerEntry->bgw_extra, sizeof(dsm_handle));
+ seg = dsm_attach(handle);
+ if (seg == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("unable to map dynamic shared memory segment")));
+
+ toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg));
+ if (toc == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("bad magic number in dynamic shared memory segment")));
+
+ before_shmem_exit(parallel_apply_shutdown, PointerGetDatum(seg));
+
+ /* Look up the shared information. */
+ shared = shm_toc_lookup(toc, PARALLEL_APPLY_KEY_SHARED, false);
+ MyParallelShared = shared;
+
+ /*
+ * Attach to the message queue.
+ */
+ mq = shm_toc_lookup(toc, PARALLEL_APPLY_KEY_MQ, false);
+ shm_mq_set_receiver(mq, MyProc);
+ mqh = shm_mq_attach(mq, seg, NULL);
+
+ /*
+ * Primary initialization is complete. Now, we can attach to our slot. This
+ * is to ensure that the leader apply worker does not write data to the
+ * uninitialized memory queue.
+ */
+ logicalrep_worker_attach(worker_slot);
+ MyParallelShared->logicalrep_worker_generation = MyLogicalRepWorker->generation;
+ MyParallelShared->logicalrep_worker_slot_no = worker_slot;
+
+ /*
+ * Attach to the error queue.
+ */
+ mq = shm_toc_lookup(toc, PARALLEL_APPLY_KEY_ERROR_QUEUE, false);
+ shm_mq_set_sender(mq, MyProc);
+ error_mqh = shm_mq_attach(mq, seg, NULL);
+
+ pq_redirect_to_shm_mq(seg, error_mqh);
+ pq_set_parallel_leader(MyLogicalRepWorker->apply_leader_pid,
+ InvalidBackendId);
+
+ MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
+ MyLogicalRepWorker->reply_time = 0;
+
+ InitializeApplyWorker();
+
+ /*
+ * Setup callback for syscache so that we know when something changes in
+ * the subscription relation state.
+ */
+ CacheRegisterSyscacheCallback(SUBSCRIPTIONRELMAP,
+ invalidate_syncing_table_states,
+ (Datum) 0);
+
+ /*
+ * Allocate the origin name in long-lived context for error context
+ * message.
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid);
+ apply_error_callback_arg.origin_name = MemoryContextStrdup(ApplyContext,
+ originname);
+
+ elog(DEBUG1, "[Parallel Apply Worker #%u] started", shared->worker_id);
+
+ LogicalParallelApplyLoop(mqh, shared);
+
+ proc_exit(0);
+}
+
+/*
+ * Handle receipt of an interrupt indicating a parallel apply worker message.
+ *
+ * Note: this is called within a signal handler! All we can do is set a flag
+ * that will cause the next CHECK_FOR_INTERRUPTS() to invoke
+ * HandleParallelApplyMessages().
+ */
+void
+HandleParallelApplyMessageInterrupt(void)
+{
+ InterruptPending = true;
+ ParallelApplyMessagePending = true;
+ SetLatch(MyLatch);
+}
+
+/*
+ * Handle a single protocol message received from a single parallel apply
+ * worker.
+ */
+static void
+HandleParallelApplyMessage(ParallelApplyWorkerInfo *winfo, StringInfo msg)
+{
+ char msgtype;
+
+ msgtype = pq_getmsgbyte(msg);
+
+ switch (msgtype)
+ {
+ case 'E': /* ErrorResponse */
+ {
+ ErrorData edata;
+ ErrorContextCallback *save_error_context_stack;
+
+ /* Parse ErrorResponse. */
+ pq_parse_errornotice(msg, &edata);
+
+ /* Death of a worker isn't enough justification for suicide. */
+ edata.elevel = Min(edata.elevel, ERROR);
+
+ /*
+ * If desired, add a context line to show that this is a
+ * message propagated from a parallel apply worker.
+ * Otherwise, it can sometimes be confusing to understand what
+ * actually happened.
+ */
+ if (edata.context)
+ edata.context = psprintf("%s\n%s", edata.context,
+ _("parallel apply worker"));
+ else
+ edata.context = pstrdup(_("parallel apply worker"));
+
+ /*
+ * Context beyond that should use the error context callbacks
+ * that were in effect in LogicalRepApplyLoop().
+ */
+ save_error_context_stack = error_context_stack;
+ error_context_stack = apply_error_context_stack;
+
+ ThrowErrorData(&edata);
+
+ /* Should not reach here after rethrowing an error. */
+ error_context_stack = save_error_context_stack;
+
+ break;
+ }
+
+ case 'X': /* Terminate, indicating clean exit */
+ {
+ shm_mq_detach(winfo->error_mq_handle);
+ winfo->error_mq_handle = NULL;
+ break;
+ }
+
+ /*
+ * Don't need to do anything about NoticeResponse and
+ * NotifyResponse as the logical replication worker doesn't need to
+ * send messages to the client.
+ */
+ case 'N':
+ case 'A':
+ break;
+ default:
+ {
+ elog(ERROR, "unrecognized message type received from parallel apply worker: %c (message length %d bytes)",
+ msgtype, msg->len);
+ }
+ }
+}
+
+/*
+ * Handle any queued protocol messages received from parallel apply workers.
+ */
+void
+HandleParallelApplyMessages(void)
+{
+ ListCell *lc;
+ MemoryContext oldcontext;
+
+ static MemoryContext hpm_context = NULL;
+
+ /*
+ * This is invoked from ProcessInterrupts(), and since some of the
+ * functions it calls contain CHECK_FOR_INTERRUPTS(), there is a potential
+ * for recursive calls if more signals are received while this runs. It's
+ * unclear that recursive entry would be safe, and it doesn't seem useful
+ * even if it is safe, so let's block interrupts until done.
+ */
+ HOLD_INTERRUPTS();
+
+ /*
+ * Moreover, CurrentMemoryContext might be pointing almost anywhere. We
+ * don't want to risk leaking data into long-lived contexts, so let's do
+ * our work here in a private context that we can reset on each use.
+ */
+ if (hpm_context == NULL) /* first time through? */
+ hpm_context = AllocSetContextCreate(TopMemoryContext,
+ "HandleParallelApplyMessages",
+ ALLOCSET_DEFAULT_SIZES);
+ else
+ MemoryContextReset(hpm_context);
+
+ oldcontext = MemoryContextSwitchTo(hpm_context);
+
+ ParallelApplyMessagePending = false;
+
+ foreach(lc, ParallelApplyWorkersList)
+ {
+ shm_mq_result res;
+ Size nbytes;
+ void *data;
+ ParallelApplyWorkerInfo *winfo = (ParallelApplyWorkerInfo *) lfirst(lc);
+
+ if (winfo->error_mq_handle == NULL)
+ continue;
+
+ res = shm_mq_receive(winfo->error_mq_handle, &nbytes,
+ &data, true);
+
+ if (res == SHM_MQ_WOULD_BLOCK)
+ break;
+ else if (res == SHM_MQ_SUCCESS)
+ {
+ StringInfoData msg;
+
+ initStringInfo(&msg);
+ appendBinaryStringInfo(&msg, data, nbytes);
+ HandleParallelApplyMessage(winfo, &msg);
+ pfree(msg.data);
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("lost connection to the leader apply worker")));
+ }
+
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Might as well clear the context on our way out */
+ MemoryContextReset(hpm_context);
+
+ RESUME_INTERRUPTS();
+}
+
+/*
+ * Set up a dynamic shared memory segment.
+ *
+ * We set up a control region that contains a ParallelApplyWorkerShared, plus
+ * one region to each of message queue and error queue.
+ *
+ * Returns true on success, false on failure.
+ */
+static bool
+parallel_apply_setup_dsm(ParallelApplyWorkerInfo *winfo)
+{
+ shm_toc_estimator e;
+ Size segsize;
+ dsm_segment *seg;
+ shm_toc *toc;
+ ParallelApplyWorkerShared *shared;
+ shm_mq *mq;
+ Size queue_size = DSM_QUEUE_SIZE;
+ Size error_queue_size = DSM_ERROR_QUEUE_SIZE;
+ int server_version;
+
+ /*
+ * Estimate how much shared memory we need.
+ *
+ * Because the TOC machinery may choose to insert padding of oddly-sized
+ * requests, we must estimate each chunk separately.
+ *
+ * We need one key to register the location of the header, and we need
+ * another key to track the location of the message queue.
+ */
+ shm_toc_initialize_estimator(&e);
+ shm_toc_estimate_chunk(&e, sizeof(ParallelApplyWorkerShared));
+ shm_toc_estimate_chunk(&e, queue_size);
+ shm_toc_estimate_chunk(&e, error_queue_size);
+
+ shm_toc_estimate_keys(&e, 3);
+ segsize = shm_toc_estimate(&e);
+
+ /* Create the shared memory segment and establish a table of contents. */
+ seg = dsm_create(shm_toc_estimate(&e), 0);
+ if (seg == NULL)
+ return false;
+
+ toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg),
+ segsize);
+
+ /* Set up the header region. */
+ shared = shm_toc_allocate(toc, sizeof(ParallelApplyWorkerShared));
+ SpinLockInit(&shared->mutex);
+
+ server_version = walrcv_server_version(LogRepWorkerWalRcvConn);
+ shared->proto_version =
+ server_version >= 160000 ? LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM :
+ server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM :
+ server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM :
+ LOGICALREP_PROTO_VERSION_NUM;
+
+ shared->in_use = false;
+ shared->worker_id = list_length(ParallelApplyWorkersList) + 1;
+
+ shm_toc_insert(toc, PARALLEL_APPLY_KEY_SHARED, shared);
+
+ /* Set up message queue for the worker. */
+ mq = shm_mq_create(shm_toc_allocate(toc, queue_size), queue_size);
+ shm_toc_insert(toc, PARALLEL_APPLY_KEY_MQ, mq);
+ shm_mq_set_sender(mq, MyProc);
+
+ /* Attach the queue. */
+ winfo->mq_handle = shm_mq_attach(mq, seg, NULL);
+
+ /* Set up error queue for the worker. */
+ mq = shm_mq_create(shm_toc_allocate(toc, error_queue_size),
+ error_queue_size);
+ shm_toc_insert(toc, PARALLEL_APPLY_KEY_ERROR_QUEUE, mq);
+ shm_mq_set_receiver(mq, MyProc);
+
+ /* Attach the queue. */
+ winfo->error_mq_handle = shm_mq_attach(mq, seg, NULL);
+
+ /* Return results to caller. */
+ winfo->dsm_seg = seg;
+ winfo->shared = shared;
+
+ return true;
+}
+
+/*
+ * Start parallel apply worker process and allocate shared memory for it.
+ */
+static ParallelApplyWorkerInfo *
+parallel_apply_setup_worker(void)
+{
+ MemoryContext oldcontext;
+ bool launched;
+ ParallelApplyWorkerInfo *winfo;
+ int napplyworkers;
+
+ elog(DEBUG1, "setting up parallel apply worker %u",
+ list_length(ParallelApplyWorkersList) + 1);
+
+ /* Check if there are free worker slot(s). */
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+ napplyworkers = logicalrep_parallel_apply_worker_count(MyLogicalRepWorker->subid);
+ LWLockRelease(LogicalRepWorkerLock);
+
+ if (napplyworkers >= max_parallel_apply_workers_per_subscription)
+ return NULL;
+
+ oldcontext = MemoryContextSwitchTo(ApplyContext);
+
+ winfo = (ParallelApplyWorkerInfo *) palloc0(sizeof(ParallelApplyWorkerInfo));
+
+ /* Setup shared memory. */
+ if (!parallel_apply_setup_dsm(winfo))
+ {
+ MemoryContextSwitchTo(oldcontext);
+ pfree(winfo);
+
+ return NULL;
+ }
+
+ launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid,
+ MySubscription->oid,
+ MySubscription->name,
+ MyLogicalRepWorker->userid,
+ InvalidOid,
+ dsm_segment_handle(winfo->dsm_seg));
+
+ if (launched)
+ ParallelApplyWorkersList = lappend(ParallelApplyWorkersList, winfo);
+ else
+ {
+ shm_mq_detach(winfo->mq_handle);
+ shm_mq_detach(winfo->error_mq_handle);
+ dsm_detach(winfo->dsm_seg);
+ pfree(winfo);
+
+ winfo = NULL;
+ }
+
+ MemoryContextSwitchTo(oldcontext);
+
+ return winfo;
+}
+
+/*
+ * Send the data to the specified parallel apply worker via shared-memory queue.
+ */
+void
+parallel_apply_send_data(ParallelApplyWorkerInfo *winfo, Size nbytes, const void *data)
+{
+ shm_mq_result result;
+
+ result = shm_mq_send(winfo->mq_handle, nbytes, data, false, true);
+
+ if (result != SHM_MQ_SUCCESS)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("could not send data to shared-memory queue")));
+}
+
+/*
+ * Wait until the parallel apply worker is free (i.e. in_use flag is false).
+ */
+void
+parallel_apply_wait_for_free(ParallelApplyWorkerInfo *winfo)
+{
+ for (;;)
+ {
+ bool in_use;
+
+ SpinLockAcquire(&winfo->shared->mutex);
+ in_use = winfo->shared->in_use;
+ SpinLockRelease(&winfo->shared->mutex);
+
+ if (!in_use)
+ break;
+
+ /* If any workers have died, we have failed. */
+ if (winfo->error_mq_handle == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("lost connection to parallel apply worker")));
+
+ /* Wait to be signalled. */
+ (void) WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ 1000L,
+ WAIT_EVENT_LOGICAL_PARALLEL_APPLY_STATE_CHANGE);
+
+ /* Reset the latch so we don't spin. */
+ ResetLatch(MyLatch);
+
+ /* An interrupt may have occurred while we were waiting. */
+ CHECK_FOR_INTERRUPTS();
+ }
+}
+
+/*
+ * Mark the current parallel apply worker as idle.
+ */
+void
+parallel_apply_set_idle(void)
+{
+ Assert(am_parallel_apply_worker());
+
+ SpinLockAcquire(&MyParallelShared->mutex);
+ MyParallelShared->in_use = false;
+ SpinLockRelease(&MyParallelShared->mutex);
+}
+
+/*
+ * Define a savepoint for a subxact in parallel apply worker if needed.
+ *
+ * The parallel apply worker can figure out if a new subtransaction was
+ * started by checking if the new change arrived with different xid. In that
+ * case define a named savepoint, so that we are able to commit/rollback it
+ * separately later.
+ */
+void
+parallel_apply_subxact_info_add(TransactionId current_xid)
+{
+ if (current_xid != stream_xid &&
+ !list_member_xid(subxactlist, (int) current_xid))
+ {
+ MemoryContext oldctx;
+ char spname[MAXPGPATH];
+
+ parallel_apply_savepoint_name(MySubscription->oid, current_xid,
+ spname, sizeof(spname));
+
+ elog(DEBUG1, "[Parallel Apply Worker #%u] defining savepoint %s",
+ MyParallelShared->worker_id, spname);
+
+ /* We must be in transaction block to define the SAVEPOINT. */
+ if (!IsTransactionBlock())
+ {
+ BeginTransactionBlock();
+ CommitTransactionCommand();
+ }
+
+ DefineSavepoint(spname);
+
+ /*
+ * CommitTransactionCommand is needed to start a subtransaction after
+ * issuing a SAVEPOINT inside a transaction block (see
+ * StartSubTransaction()).
+ */
+ CommitTransactionCommand();
+
+ oldctx = MemoryContextSwitchTo(ApplyContext);
+ subxactlist = lappend_xid(subxactlist, current_xid);
+ MemoryContextSwitchTo(oldctx);
+ }
+}
+
+/*
+ * Handle STREAM ABORT message when the transaction was applied in a parallel
+ * apply worker.
+ */
+void
+parallel_apply_stream_abort(LogicalRepStreamAbortData *abort_data,
+ bool read_abort_lsn)
+{
+ TransactionId xid = abort_data->xid;
+ TransactionId subxid = abort_data->subxid;
+
+ elog(DEBUG1, "[Parallel Apply Worker #%u] aborting current transaction xid=%u, subxid=%u",
+ MyParallelShared->worker_id , GetCurrentTransactionIdIfAny(),
+ GetCurrentSubTransactionId());
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ if (read_abort_lsn)
+ {
+ replorigin_session_origin_lsn = abort_data->abort_lsn;
+ replorigin_session_origin_timestamp = abort_data->abort_time;
+ }
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact,
+ * so just free the subxactlist.
+ */
+ if (subxid == xid)
+ {
+ parallel_apply_replorigin_setup();
+
+ AbortCurrentTransaction();
+
+ if (IsTransactionBlock())
+ {
+ EndTransactionBlock(false);
+ CommitTransactionCommand();
+ }
+
+ parallel_apply_replorigin_reset();
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ list_free(subxactlist);
+ subxactlist = NIL;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. Rollback to the savepoint.
+ *
+ * We also need to read the subxactlist, determine the offset
+ * tracked for the subxact, and truncate the list.
+ */
+ int i;
+ bool found = false;
+ char spname[MAXPGPATH];
+
+ parallel_apply_savepoint_name(MySubscription->oid, subxid, spname,
+ sizeof(spname));
+
+ elog(DEBUG1, "[Parallel Apply Worker #%u] rolling back to savepoint %s",
+ MyParallelShared->worker_id, spname);
+
+ for (i = list_length(subxactlist) - 1; i >= 0; i--)
+ {
+ xid = (TransactionId) list_nth_xid(subxactlist, i);
+ if (xid == subxid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (found)
+ {
+ RollbackToSavepoint(spname);
+ CommitTransactionCommand();
+ subxactlist = list_truncate(subxactlist, i + 1);
+ }
+
+ pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL);
+ }
+
+ parallel_apply_set_idle();
+}
+
+/*
+ * Form the savepoint name for the streaming transaction.
+ *
+ * Return the name in the supplied buffer.
+ */
+void
+parallel_apply_savepoint_name(Oid suboid, TransactionId xid,
+ char *spname, int szsp)
+{
+ snprintf(spname, szsp, "pg_sp_%u_%u", suboid, xid);
+}
+
+/* Setup replication origin tracking. */
+void
+parallel_apply_replorigin_setup(void)
+{
+ RepOriginId originid;
+ char originname[NAMEDATALEN];
+ bool started_tx = false;
+
+ /* This function might be called inside or outside of transaction. */
+ if (!IsTransactionState())
+ {
+ StartTransactionCommand();
+ started_tx = true;
+ }
+
+ snprintf(originname, sizeof(originname), "pg_%u", MySubscription->oid);
+ originid = replorigin_by_name(originname, false);
+ replorigin_session_setup(originid);
+ replorigin_session_origin = originid;
+
+ if (started_tx)
+ CommitTransactionCommand();
+}
+
+/* Reset replication origin tracking. */
+void
+parallel_apply_replorigin_reset(void)
+{
+ bool started_tx = false;
+
+ /* This function might be called inside or outside of transaction. */
+ if (!IsTransactionState())
+ {
+ StartTransactionCommand();
+ started_tx = true;
+ }
+
+ replorigin_session_reset();
+
+ replorigin_session_origin = InvalidRepOriginId;
+ replorigin_session_origin_lsn = InvalidXLogRecPtr;
+ replorigin_session_origin_timestamp = 0;
+
+ if (started_tx)
+ CommitTransactionCommand();
+}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 1667d72..a437683 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -652,9 +652,10 @@ DecodeCommit(LogicalDecodingContext *ctx, XLogRecordBuffer *buf,
{
for (i = 0; i < parsed->nsubxacts; i++)
{
- ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr);
+ ReorderBufferForget(ctx->reorder, parsed->subxacts[i], buf->origptr,
+ commit_time);
}
- ReorderBufferForget(ctx->reorder, xid, buf->origptr);
+ ReorderBufferForget(ctx->reorder, xid, buf->origptr, commit_time);
return;
}
@@ -822,10 +823,11 @@ DecodeAbort(LogicalDecodingContext *ctx, XLogRecordBuffer *buf,
for (i = 0; i < parsed->nsubxacts; i++)
{
ReorderBufferAbort(ctx->reorder, parsed->subxacts[i],
- buf->record->EndRecPtr);
+ buf->record->EndRecPtr, abort_time);
}
- ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr);
+ ReorderBufferAbort(ctx->reorder, xid, buf->record->EndRecPtr,
+ abort_time);
}
/* update the decoding stats */
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 3bbd522..7a46cc2 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -54,6 +54,7 @@
int max_logical_replication_workers = 4;
int max_sync_workers_per_subscription = 2;
+int max_parallel_apply_workers_per_subscription = 2;
LogicalRepWorker *MyLogicalRepWorker = NULL;
@@ -73,6 +74,7 @@ static void logicalrep_launcher_onexit(int code, Datum arg);
static void logicalrep_worker_onexit(int code, Datum arg);
static void logicalrep_worker_detach(void);
static void logicalrep_worker_cleanup(LogicalRepWorker *worker);
+static void logicalrep_worker_stop_internal(LogicalRepWorker *worker);
static bool on_commit_launcher_wakeup = false;
@@ -151,8 +153,10 @@ get_subscription_list(void)
*
* This is only needed for cleaning up the shared memory in case the worker
* fails to attach.
+ *
+ * Return whether the attach was successful.
*/
-static void
+static bool
WaitForReplicationWorkerAttach(LogicalRepWorker *worker,
uint16 generation,
BackgroundWorkerHandle *handle)
@@ -168,11 +172,11 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker,
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- /* Worker either died or has started; no need to do anything. */
+ /* Worker either died or has started. Return false if died. */
if (!worker->in_use || worker->proc)
{
LWLockRelease(LogicalRepWorkerLock);
- return;
+ return worker->in_use;
}
LWLockRelease(LogicalRepWorkerLock);
@@ -187,7 +191,7 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker,
if (generation == worker->generation)
logicalrep_worker_cleanup(worker);
LWLockRelease(LogicalRepWorkerLock);
- return;
+ return false;
}
/*
@@ -209,6 +213,8 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker,
/*
* Walks the workers array and searches for one that matches given
* subscription id and relid.
+ *
+ * We are only interested in the leader apply worker or table sync worker.
*/
LogicalRepWorker *
logicalrep_worker_find(Oid subid, Oid relid, bool only_running)
@@ -223,6 +229,10 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running)
{
LogicalRepWorker *w = &LogicalRepCtx->workers[i];
+ /* Skip parallel apply workers. */
+ if (isParallelApplyWorker(w))
+ continue;
+
if (w->in_use && w->subid == subid && w->relid == relid &&
(!only_running || w->proc))
{
@@ -259,11 +269,13 @@ logicalrep_workers_find(Oid subid, bool only_running)
}
/*
- * Start new apply background worker, if possible.
+ * Start new background worker, if possible.
+ *
+ * Returns true on success, false on failure.
*/
-void
+bool
logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid,
- Oid relid)
+ Oid relid, dsm_handle subworker_dsm)
{
BackgroundWorker bgw;
BackgroundWorkerHandle *bgw_handle;
@@ -272,7 +284,12 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid,
int slot = 0;
LogicalRepWorker *worker = NULL;
int nsyncworkers;
+ int nparallelapplyworkers;
TimestampTz now;
+ bool is_subworker = (subworker_dsm != DSM_HANDLE_INVALID);
+
+ /* Sanity check: we don't support table sync in subworker. */
+ Assert(!(is_subworker && OidIsValid(relid)));
ereport(DEBUG1,
(errmsg_internal("starting logical replication worker for subscription \"%s\"",
@@ -350,7 +367,19 @@ retry:
if (OidIsValid(relid) && nsyncworkers >= max_sync_workers_per_subscription)
{
LWLockRelease(LogicalRepWorkerLock);
- return;
+ return false;
+ }
+
+ nparallelapplyworkers = logicalrep_parallel_apply_worker_count(subid);
+
+ /*
+ * Return silently if the number of parallel apply workers reached the
+ * limit per subscription.
+ */
+ if (is_subworker && nparallelapplyworkers >= max_parallel_apply_workers_per_subscription)
+ {
+ LWLockRelease(LogicalRepWorkerLock);
+ return false;
}
/*
@@ -364,7 +393,7 @@ retry:
(errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
errmsg("out of logical replication worker slots"),
errhint("You might need to increase max_logical_replication_workers.")));
- return;
+ return false;
}
/* Prepare the worker slot. */
@@ -379,6 +408,7 @@ retry:
worker->relstate = SUBREL_STATE_UNKNOWN;
worker->relstate_lsn = InvalidXLogRecPtr;
worker->stream_fileset = NULL;
+ worker->apply_leader_pid = is_subworker ? MyProcPid : InvalidPid;
worker->last_lsn = InvalidXLogRecPtr;
TIMESTAMP_NOBEGIN(worker->last_send_time);
TIMESTAMP_NOBEGIN(worker->last_recv_time);
@@ -396,10 +426,18 @@ retry:
BGWORKER_BACKEND_DATABASE_CONNECTION;
bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
- snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain");
+
+ if (is_subworker)
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ParallelApplyWorkerMain");
+ else
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain");
+
if (OidIsValid(relid))
snprintf(bgw.bgw_name, BGW_MAXLEN,
"logical replication worker for subscription %u sync %u", subid, relid);
+ else if (is_subworker)
+ snprintf(bgw.bgw_name, BGW_MAXLEN,
+ "logical replication parallel apply worker for subscription %u", subid);
else
snprintf(bgw.bgw_name, BGW_MAXLEN,
"logical replication worker for subscription %u", subid);
@@ -409,6 +447,9 @@ retry:
bgw.bgw_notify_pid = MyProcPid;
bgw.bgw_main_arg = Int32GetDatum(slot);
+ if (is_subworker)
+ memcpy(bgw.bgw_extra, &subworker_dsm, sizeof(dsm_handle));
+
if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle))
{
/* Failed to start worker, so clean up the worker slot. */
@@ -421,11 +462,11 @@ retry:
(errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
errmsg("out of background worker slots"),
errhint("You might need to increase max_worker_processes.")));
- return;
+ return false;
}
/* Now wait until it attaches. */
- WaitForReplicationWorkerAttach(worker, generation, bgw_handle);
+ return WaitForReplicationWorkerAttach(worker, generation, bgw_handle);
}
/*
@@ -436,19 +477,52 @@ void
logicalrep_worker_stop(Oid subid, Oid relid)
{
LogicalRepWorker *worker;
- uint16 generation;
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
worker = logicalrep_worker_find(subid, relid, false);
- /* No worker, nothing to do. */
- if (!worker)
+ if (worker)
{
- LWLockRelease(LogicalRepWorkerLock);
- return;
+ Assert(!isParallelApplyWorker(worker));
+ logicalrep_worker_stop_internal(worker);
}
+ LWLockRelease(LogicalRepWorkerLock);
+}
+
+/*
+ * Stop the logical replication worker corresponding to the input slot number,
+ * and wait until it detaches from the slot.
+ */
+void
+logicalrep_worker_stop_by_slot(int slot_no, uint16 generation)
+{
+ LogicalRepWorker *worker = &LogicalRepCtx->workers[slot_no];
+
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+
+ /* Return if the generation doesn't match or the worker is not alive. */
+ if (worker->generation != generation ||
+ worker->proc == NULL)
+ return;
+
+ logicalrep_worker_stop_internal(worker);
+
+ LWLockRelease(LogicalRepWorkerLock);
+}
+
+/*
+ * Workhorse for logicalrep_worker_stop() and logicalrep_worker_detach(). Stop
+ * the worker and wait for it to die.
+ */
+static void
+logicalrep_worker_stop_internal(LogicalRepWorker *worker)
+{
+ uint16 generation;
+
+ Assert(LWLockHeldByMe(LogicalRepWorkerLock));
+
/*
* Remember which generation was our worker so we can check if what we see
* is still the same one.
@@ -485,10 +559,7 @@ logicalrep_worker_stop(Oid subid, Oid relid)
* different, meaning that a different worker has taken the slot.
*/
if (!worker->in_use || worker->generation != generation)
- {
- LWLockRelease(LogicalRepWorkerLock);
return;
- }
/* Worker has assigned proc, so it has started. */
if (worker->proc)
@@ -522,8 +593,6 @@ logicalrep_worker_stop(Oid subid, Oid relid)
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
}
-
- LWLockRelease(LogicalRepWorkerLock);
}
/*
@@ -599,6 +668,29 @@ logicalrep_worker_attach(int slot)
static void
logicalrep_worker_detach(void)
{
+ /*
+ * This is the leader apply worker; stop all the parallel apply workers
+ * previously started from here.
+ */
+ if (!isParallelApplyWorker(MyLogicalRepWorker))
+ {
+ List *workers;
+ ListCell *lc;
+
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+
+ workers = logicalrep_workers_find(MyLogicalRepWorker->subid, true);
+ foreach(lc, workers)
+ {
+ LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc);
+
+ if (isParallelApplyWorker(w))
+ logicalrep_worker_stop_internal(w);
+ }
+
+ LWLockRelease(LogicalRepWorkerLock);
+ }
+
/* Block concurrent access. */
LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE);
@@ -621,6 +713,7 @@ logicalrep_worker_cleanup(LogicalRepWorker *worker)
worker->userid = InvalidOid;
worker->subid = InvalidOid;
worker->relid = InvalidOid;
+ worker->apply_leader_pid = InvalidPid;
}
/*
@@ -680,6 +773,33 @@ logicalrep_sync_worker_count(Oid subid)
}
/*
+ * Count the number of registered (but not necessarily running) parallel apply
+ * workers for a subscription.
+ */
+int
+logicalrep_parallel_apply_worker_count(Oid subid)
+{
+ int i;
+ int res = 0;
+
+ Assert(LWLockHeldByMe(LogicalRepWorkerLock));
+
+ /*
+ * Scan all attached parallel apply workers, only counting those which
+ * have the given subscription id.
+ */
+ for (i = 0; i < max_logical_replication_workers; i++)
+ {
+ LogicalRepWorker *w = &LogicalRepCtx->workers[i];
+
+ if (w->subid == subid && isParallelApplyWorker(w))
+ res++;
+ }
+
+ return res;
+}
+
+/*
* ApplyLauncherShmemSize
* Compute space needed for replication launcher shared memory
*/
@@ -868,7 +988,7 @@ ApplyLauncherMain(Datum main_arg)
wait_time = wal_retrieve_retry_interval;
logicalrep_worker_launch(sub->dbid, sub->oid, sub->name,
- sub->owner, InvalidOid);
+ sub->owner, InvalidOid, DSM_HANDLE_INVALID);
}
}
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index ff8513e..47bd811 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -1163,31 +1163,56 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
/*
* Write STREAM ABORT to the output stream. Note that xid and subxid will be
* same for the top-level transaction abort.
+ *
+ * If write_abort_lsn is true, send the abort_lsn and abort_time fields,
+ * otherwise don't.
*/
void
logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
- TransactionId subxid)
+ ReorderBufferTXN *txn, XLogRecPtr abort_lsn,
+ bool write_abort_lsn)
{
pq_sendbyte(out, LOGICAL_REP_MSG_STREAM_ABORT);
- Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(txn->xid));
/* transaction ID */
pq_sendint32(out, xid);
- pq_sendint32(out, subxid);
+ pq_sendint32(out, txn->xid);
+
+ if (write_abort_lsn)
+ {
+ pq_sendint64(out, abort_lsn);
+ pq_sendint64(out, txn->xact_time.abort_time);
+ }
}
/*
* Read STREAM ABORT from the output stream.
+ *
+ * If read_abort_lsn is true, try to read the abort_lsn and abort_time fields,
+ * otherwise don't.
*/
void
-logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
- TransactionId *subxid)
+logicalrep_read_stream_abort(StringInfo in,
+ LogicalRepStreamAbortData *abort_data,
+ bool read_abort_lsn)
{
- Assert(xid && subxid);
+ Assert(abort_data);
- *xid = pq_getmsgint(in, 4);
- *subxid = pq_getmsgint(in, 4);
+ abort_data->xid = pq_getmsgint(in, 4);
+ abort_data->subxid = pq_getmsgint(in, 4);
+
+ if (read_abort_lsn)
+ {
+ abort_data->abort_lsn = pq_getmsgint64(in);
+ abort_data->abort_time = pq_getmsgint64(in);
+ }
+ else
+ {
+ abort_data->abort_lsn = InvalidXLogRecPtr;
+ abort_data->abort_time = 0;
+ }
}
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 89cf9f9..5a20c74 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2839,7 +2839,8 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid,
* disk.
*/
void
-ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
+ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
+ TimestampTz abort_time)
{
ReorderBufferTXN *txn;
@@ -2850,6 +2851,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ txn->xact_time.abort_time = abort_time;
+
/* For streamed transactions notify the remote node about the abort. */
if (rbtxn_is_streamed(txn))
{
@@ -2924,7 +2927,8 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* to this xid might re-create the transaction incompletely.
*/
void
-ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
+ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
+ TimestampTz abort_time)
{
ReorderBufferTXN *txn;
@@ -2935,6 +2939,8 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ txn->xact_time.abort_time = abort_time;
+
/* For streamed transactions notify the remote node about the abort. */
if (rbtxn_is_streamed(txn))
rb->stream_abort(rb, txn, lsn);
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index d37d8a0..60dfb86 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -568,7 +568,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn)
MySubscription->oid,
MySubscription->name,
MyLogicalRepWorker->userid,
- rstate->relid);
+ rstate->relid,
+ DSM_HANDLE_INVALID);
hentry->last_start_time = now;
}
}
@@ -589,6 +590,9 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn)
void
process_syncing_tables(XLogRecPtr current_lsn)
{
+ if (am_parallel_apply_worker())
+ return;
+
if (am_tablesync_worker())
process_syncing_tables_for_sync(current_lsn);
else
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 5f8c541..c191797 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -22,8 +22,49 @@
* STREAMED TRANSACTIONS
* ---------------------
* Streamed transactions (large transactions exceeding a memory limit on the
- * upstream) are not applied immediately, but instead, the data is written
- * to temporary files and then applied at once when the final commit arrives.
+ * upstream) are applied using one of two approaches:
+ *
+ * 1) Parallel apply workers
+ *
+ * If streaming = parallel, we assign a new parallel apply worker (if
+ * available) as soon as the xact's first stream is received. The leader apply
+ * worker will send changes to this new worker via shared memory. We keep this
+ * worker assigned till the transaction commit is received and also wait for
+ * the worker to finish at commit. This preserves commit ordering and avoids
+ * file I/O in most cases. We still need to spill to a file if there is no
+ * worker available. It is important to maintain commit order to avoid failures
+ * due to (a) transaction dependencies, say if we insert a row in the first
+ * transaction and update it in the second transaction then allowing to apply
+ * both in parallel can lead to failure in the update. (b) deadlocks, allowing
+ * transactions that update the same set of rows/tables in opposite order to be
+ * applied in parallel can lead to deadlocks.
+ *
+ * If there are enough parallel apply workers (reached half of the
+ * max_parallel_apply_workers_per_subscription) in the pool, the new parallel apply
+ * worker will be shut down at transaction end. Otherwise, it will stay in the
+ * pool.
+ *
+ * The leader apply worker will create separate dynamic shared memory segment
+ * when each parallel apply worker starts. The reason for this design is that
+ * we cannot count how many workers will be started. It may be possible to
+ * allocate enough shared memory in one segment based on the maximum number of
+ * parallel apply workers (max_parallel_apply_workers_per_subscription), but this
+ * may waste some memory if no process is actually started.
+ *
+ * The dynamic shared memory segment will contain (1) a shm_mq that can be used
+ * to transport errors (and other messages reported via elog/ereport) from the
+ * parallel apply worker to leader apply worker (2) another shm_mq that can
+ * be used to send changes in the transaction from leader apply worker to
+ * parallel apply worker (3) necessary information to be shared among parallel
+ * apply workers and leader apply worker(i.e. protocol version, streaming
+ * transaction ID, in_use flag and the corresponding LogicalRepWorker slot
+ * information).
+ *
+ * 2) Write to temporary files and apply when the final commit arrives
+ *
+ * If no worker is available to handle the streamed transaction, the data is
+ * written to temporary files and then applied at once when the final commit
+ * arrives.
*
* Unlike the regular (non-streamed) case, handling streamed transactions has
* to handle aborts of both the toplevel transaction and subtransactions. This
@@ -219,20 +260,35 @@ typedef struct ApplyExecutionData
PartitionTupleRouting *proute; /* partition routing info */
} ApplyExecutionData;
-/* Struct for saving and restoring apply errcontext information */
-typedef struct ApplyErrorCallbackArg
+/*
+ * What action to take for the transaction.
+ *
+ * TA_APPLY_IN_LEADER_WORKER means that we are in the leader apply worker and
+ * changes of the transaction are applied directly in the worker.
+ *
+ * TA_SERIALIZE_TO_FILE means that we are in leader apply worker and changes
+ * are written to temporary files and then applied when the final commit
+ * arrives.
+ *
+ * TA_APPLY_IN_PARALLEL_WORKER means that we are in the parallel apply worker
+ * and changes of the transaction are applied directly in the worker.
+ *
+ * TA_SEND_TO_PARALLEL_WORKER means that we are in the leader apply worker and
+ * need to send the changes to the parallel apply worker.
+ */
+typedef enum
{
- LogicalRepMsgType command; /* 0 if invalid */
- LogicalRepRelMapEntry *rel;
+ /* The action for non-streaming transactions. */
+ TA_APPLY_IN_LEADER_WORKER,
- /* Remote node information */
- int remote_attnum; /* -1 if invalid */
- TransactionId remote_xid;
- XLogRecPtr finish_lsn;
- char *origin_name;
-} ApplyErrorCallbackArg;
+ /* Actions for streaming transactions. */
+ TA_SERIALIZE_TO_FILE,
+ TA_APPLY_IN_PARALLEL_WORKER,
+ TA_SEND_TO_PARALLEL_WORKER
+} TransactionApplyAction;
-static ApplyErrorCallbackArg apply_error_callback_arg =
+/* errcontext tracker */
+ApplyErrorCallbackArg apply_error_callback_arg =
{
.command = 0,
.rel = NULL,
@@ -242,7 +298,9 @@ static ApplyErrorCallbackArg apply_error_callback_arg =
.origin_name = NULL,
};
-static MemoryContext ApplyMessageContext = NULL;
+ErrorContextCallback *apply_error_context_stack = NULL;
+
+MemoryContext ApplyMessageContext = NULL;
MemoryContext ApplyContext = NULL;
/* per stream context for streaming transactions */
@@ -251,27 +309,35 @@ static MemoryContext LogicalStreamingContext = NULL;
WalReceiverConn *LogRepWorkerWalRcvConn = NULL;
Subscription *MySubscription = NULL;
-static bool MySubscriptionValid = false;
+bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
/* fields valid only when processing streamed transaction */
-static bool in_streamed_transaction = false;
+bool in_streamed_transaction = false;
+
+TransactionId stream_xid = InvalidTransactionId;
-static TransactionId stream_xid = InvalidTransactionId;
+/*
+ * The number of changes sent to parallel apply workers during one streaming
+ * block.
+ */
+static uint32 nchanges = 0;
/*
* We enable skipping all data modification changes (INSERT, UPDATE, etc.) for
* the subscription if the remote transaction's finish LSN matches the subskiplsn.
* Once we start skipping changes, we don't stop it until we skip all changes of
* the transaction even if pg_subscription is updated and MySubscription->skiplsn
- * gets changed or reset during that. Also, in streaming transaction cases, we
- * don't skip receiving and spooling the changes since we decide whether or not
+ * gets changed or reset during that. Also, in streaming transaction cases (streaming = on),
+ * we don't skip receiving and spooling the changes since we decide whether or not
* to skip applying the changes when starting to apply changes. The subskiplsn is
* cleared after successfully skipping the transaction or applying non-empty
* transaction. The latter prevents the mistakenly specified subskiplsn from
- * being left.
+ * being left. Note that we cannot skip the streaming transactions when using
+ * parallel apply workers because we cannot get the finish LSN before
+ * applying the changes.
*/
static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr;
#define is_skipping_changes() (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn)))
@@ -324,9 +390,6 @@ static void maybe_reread_subscription(void);
static void DisableSubscriptionAndExit(void);
-/* prototype needed because of stream_commit */
-static void apply_dispatch(StringInfo s);
-
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
@@ -359,28 +422,52 @@ static void stop_skipping_changes(void);
static void clear_subscription_skip_lsn(XLogRecPtr finish_lsn);
/* Functions for apply error callback */
-static void apply_error_callback(void *arg);
static inline void set_apply_error_context_xact(TransactionId xid, XLogRecPtr lsn);
static inline void reset_apply_error_context_info(void);
+static TransactionApplyAction get_transaction_apply_action(TransactionId xid,
+ ParallelApplyWorkerInfo **winfo);
+
/*
* Should this worker apply changes for given relation.
*
* This is mainly needed for initial relation data sync as that runs in
* separate worker process running in parallel and we need some way to skip
- * changes coming to the main apply worker during the sync of a table.
+ * changes coming to the leader apply worker during the sync of a table.
*
* Note we need to do smaller or equals comparison for SYNCDONE state because
* it might hold position of end of initial slot consistent point WAL
* record + 1 (ie start of next record) and next record can be COMMIT of
* transaction we are now processing (which is what we set remote_final_lsn
* to in apply_handle_begin).
+ *
+ * Note that for streaming transactions that are being applied in parallel
+ * apply worker, we disallow applying changes on a table that is not in
+ * the READY state, because we cannot decide whether to apply the change as we
+ * won't know remote_final_lsn by that time.
+ *
+ * We already checked this in parallel_apply_can_start() before assigning the
+ * streaming transaction to the parallel worker, but it also needs to be
+ * checked here because if the user executes ALTER SUBSCRIPTION ... REFRESH
+ * PUBLICATION in parallel, the new table can be added to pg_subscription_rel
+ * in parallel while applying this transaction.
*/
static bool
should_apply_changes_for_rel(LogicalRepRelMapEntry *rel)
{
if (am_tablesync_worker())
return MyLogicalRepWorker->relid == rel->localreloid;
+ else if (am_parallel_apply_worker())
+ {
+ if (rel->state != SUBREL_STATE_READY)
+ ereport(ERROR,
+ (errmsg("logical replication apply workers for subscription \"%s\" will restart",
+ MySubscription->name),
+ errdetail("Cannot handle streamed replication transaction by parallel "
+ "apply workers until all tables are synchronized.")));
+
+ return true;
+ }
else
return (rel->state == SUBREL_STATE_READY ||
(rel->state == SUBREL_STATE_SYNCDONE &&
@@ -426,43 +513,92 @@ end_replication_step(void)
}
/*
- * Handle streamed transactions.
+ * Handle streamed transactions for both the leader apply worker and the parallel
+ * apply workers.
+ *
+ * In streaming case (receiving a block of streamed transaction), for
+ * SUBSTREAM_ON mode, simply redirect it to a file for the proper toplevel
+ * transaction, and for SUBSTREAM_PARALLEL mode, send the changes to parallel
+ * apply workers (LOGICAL_REP_MSG_RELATION or LOGICAL_REP_MSG_TYPE changes
+ * will be applied by both leader apply worker and parallel apply workers).
*
- * If in streaming mode (receiving a block of streamed transaction), we
- * simply redirect it to a file for the proper toplevel transaction.
+ * For non-streamed transactions, returns false;
+ * For streamed transactions, returns true if in leader apply worker, false
+ * otherwise.
*
- * Returns true for streamed transactions, false otherwise (regular mode).
+ * Exception: If the message being processed is LOGICAL_REP_MSG_RELATION
+ * or LOGICAL_REP_MSG_TYPE, return false even if the message needs to be sent
+ * to a parallel apply worker.
*/
static bool
handle_streamed_transaction(LogicalRepMsgType action, StringInfo s)
{
- TransactionId xid;
+ TransactionId current_xid = InvalidTransactionId;
+ ParallelApplyWorkerInfo *winfo = NULL;
+ TransactionApplyAction apply_action;
+
+ apply_action = get_transaction_apply_action(stream_xid, &winfo);
/* not in streaming mode */
- if (!in_streamed_transaction)
+ if (apply_action == TA_APPLY_IN_LEADER_WORKER)
return false;
- Assert(stream_fd != NULL);
Assert(TransactionIdIsValid(stream_xid));
/*
* We should have received XID of the subxact as the first part of the
* message, so extract it.
*/
- xid = pq_getmsgint(s, 4);
+ current_xid = pq_getmsgint(s, 4);
- if (!TransactionIdIsValid(xid))
+ if (!TransactionIdIsValid(current_xid))
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
errmsg_internal("invalid transaction ID in streamed replication transaction")));
- /* Add the new subxact to the array (unless already there). */
- subxact_info_add(xid);
+ if (apply_action == TA_APPLY_IN_PARALLEL_WORKER)
+ {
+ /* Define a savepoint for a subxact if needed. */
+ parallel_apply_subxact_info_add(current_xid);
- /* write the change to the current file */
- stream_write_change(action, s);
+ return false;
+ }
+ else if (apply_action == TA_SEND_TO_PARALLEL_WORKER)
+ {
+ Assert(winfo);
+
+ parallel_apply_send_data(winfo, s->len, s->data);
+ nchanges += 1;
+
+ /*
+ * XXX The publisher side doesn't always send relation/type update
+ * messages after the streaming transaction, so also update the
+ * relation/type in leader apply worker here. See function
+ * cleanup_rel_sync_cache.
+ */
+ if (action == LOGICAL_REP_MSG_RELATION ||
+ action == LOGICAL_REP_MSG_TYPE)
+ return false;
+
+ return true;
+ }
+ else if (apply_action == TA_SERIALIZE_TO_FILE)
+ {
+ Assert(stream_fd != NULL);
- return true;
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(current_xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+ }
+ else
+ {
+ Assert(false);
+ return false; /* silence compiler warning */
+ }
}
/*
@@ -898,8 +1034,11 @@ apply_handle_prepare_internal(LogicalRepPreparedTxnData *prepare_data)
* BeginTransactionBlock is necessary to balance the EndTransactionBlock
* called within the PrepareTransactionBlock below.
*/
- BeginTransactionBlock();
- CommitTransactionCommand(); /* Completes the preceding Begin command. */
+ if (!IsTransactionBlock())
+ {
+ BeginTransactionBlock();
+ CommitTransactionCommand(); /* Completes the preceding Begin command. */
+ }
/*
* Update origin state so we can restart streaming from correct position
@@ -968,6 +1107,12 @@ apply_handle_prepare(StringInfo s)
/*
* Handle a COMMIT PREPARED of a previously PREPARED transaction.
+ *
+ * Note that we don't need to wait here if the transaction was prepared in a
+ * parallel apply worker. Because we have already waited for the prepare to
+ * finish in apply_handle_stream_prepare() which will ensure all the operations
+ * in that transaction have happened in the subscriber and no concurrent
+ * transaction can create deadlock or transaction dependency issues.
*/
static void
apply_handle_commit_prepared(StringInfo s)
@@ -1011,6 +1156,12 @@ apply_handle_commit_prepared(StringInfo s)
/*
* Handle a ROLLBACK PREPARED of a previously PREPARED TRANSACTION.
+ *
+ * Note that we don't need to wait here if the transaction was prepared in a
+ * parallel apply worker. Because we have already waited for the prepare to
+ * finish in apply_handle_stream_prepare() which will ensure all the operations
+ * in that transaction have happened in the subscriber and no concurrent
+ * transaction can create deadlock or transaction dependency issues.
*/
static void
apply_handle_rollback_prepared(StringInfo s)
@@ -1064,15 +1215,13 @@ apply_handle_rollback_prepared(StringInfo s)
/*
* Handle STREAM PREPARE.
- *
- * Logic is in two parts:
- * 1. Replay all the spooled operations
- * 2. Mark the transaction as prepared
*/
static void
apply_handle_stream_prepare(StringInfo s)
{
LogicalRepPreparedTxnData prepare_data;
+ ParallelApplyWorkerInfo *winfo = NULL;
+ TransactionApplyAction apply_action;
if (in_streamed_transaction)
ereport(ERROR,
@@ -1088,24 +1237,72 @@ apply_handle_stream_prepare(StringInfo s)
logicalrep_read_stream_prepare(s, &prepare_data);
set_apply_error_context_xact(prepare_data.xid, prepare_data.prepare_lsn);
- elog(DEBUG1, "received prepare for streamed transaction %u", prepare_data.xid);
+ apply_action = get_transaction_apply_action(prepare_data.xid, &winfo);
- /* Replay all the spooled operations. */
- apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn);
+ if (apply_action == TA_APPLY_IN_PARALLEL_WORKER)
+ {
+ parallel_apply_replorigin_setup();
- /* Mark the transaction as prepared. */
- apply_handle_prepare_internal(&prepare_data);
+ /* Mark the transaction as prepared. */
+ apply_handle_prepare_internal(&prepare_data);
- CommitTransactionCommand();
+ CommitTransactionCommand();
- pgstat_report_stat(false);
+ parallel_apply_replorigin_reset();
- store_flush_position(prepare_data.end_lsn);
+ list_free(subxactlist);
+ subxactlist = NIL;
- in_remote_transaction = false;
+ parallel_apply_set_idle();
+ }
+ else if (apply_action == TA_SEND_TO_PARALLEL_WORKER)
+ {
+ Assert(winfo);
+
+ parallel_apply_replorigin_reset();
- /* unlink the files with serialized changes and subxact info. */
- stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid);
+ /* Send STREAM PREPARE message to the parallel apply worker. */
+ parallel_apply_send_data(winfo, s->len, s->data);
+
+ /*
+ * After sending the data to the parallel apply worker, wait for
+ * that worker to finish. This is necessary to maintain commit
+ * order which avoids failures due to transaction dependencies and
+ * deadlocks.
+ */
+ parallel_apply_wait_for_free(winfo);
+ parallel_apply_replorigin_setup();
+ parallel_apply_free_worker(winfo, prepare_data.xid);
+
+ elog(DEBUG1, "streamed transaction %u was prepared using an parallel apply worker",
+ prepare_data.xid);
+
+ store_flush_position(prepare_data.end_lsn);
+ }
+ else if (apply_action == TA_SERIALIZE_TO_FILE)
+ {
+ /*
+ * The transaction has been serialized to file, so replay all the
+ * spooled operations.
+ */
+ apply_spooled_messages(prepare_data.xid, prepare_data.prepare_lsn);
+
+ /* Mark the transaction as prepared. */
+ apply_handle_prepare_internal(&prepare_data);
+
+ CommitTransactionCommand();
+
+ store_flush_position(prepare_data.end_lsn);
+
+ /* Unlink the files with serialized changes and subxact info. */
+ stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid);
+ }
+ else
+ Assert(false);
+
+ pgstat_report_stat(false);
+
+ in_remote_transaction = false;
/* Process any tables that are being synchronized in parallel. */
process_syncing_tables(prepare_data.end_lsn);
@@ -1134,7 +1331,7 @@ apply_handle_origin(StringInfo s)
* ORIGIN message can only come inside streaming transaction or inside
* remote transaction and before any actual writes.
*/
- if (!in_streamed_transaction &&
+ if (!in_streamed_transaction && stream_apply_worker == NULL &&
(!in_remote_transaction ||
(IsTransactionState() && !am_tablesync_worker())))
ereport(ERROR,
@@ -1149,24 +1346,14 @@ static void
apply_handle_stream_start(StringInfo s)
{
bool first_segment;
+ ParallelApplyWorkerInfo *winfo = NULL;
+ TransactionApplyAction apply_action;
if (in_streamed_transaction)
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
errmsg_internal("duplicate STREAM START message")));
- /*
- * Start a transaction on stream start, this transaction will be committed
- * on the stream stop unless it is a tablesync worker in which case it
- * will be committed after processing all the messages. We need the
- * transaction for handling the buffile, used for serializing the
- * streaming data and subxact info.
- */
- begin_replication_step();
-
- /* notify handle methods we're processing a remote transaction */
- in_streamed_transaction = true;
-
/* extract XID of the top-level transaction */
stream_xid = logicalrep_read_stream_start(s, &first_segment);
@@ -1178,35 +1365,88 @@ apply_handle_stream_start(StringInfo s)
set_apply_error_context_xact(stream_xid, InvalidXLogRecPtr);
/*
- * Initialize the worker's stream_fileset if we haven't yet. This will be
- * used for the entire duration of the worker so create it in a permanent
- * context. We create this on the very first streaming message from any
- * transaction and then use it for this and other streaming transactions.
- * Now, we could create a fileset at the start of the worker as well but
- * then we won't be sure that it will ever be used.
+ * For the first stream start, check if there is any free parallel apply
+ * worker we can use to process this transaction.
*/
- if (MyLogicalRepWorker->stream_fileset == NULL)
+ if (first_segment)
+ parallel_apply_start_worker(stream_xid);
+
+ apply_action = get_transaction_apply_action(stream_xid, &winfo);
+
+ if (apply_action == TA_APPLY_IN_PARALLEL_WORKER)
{
- MemoryContext oldctx;
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a
+ * remote transaction.
+ */
+ in_remote_transaction = true;
+ }
+ else if (apply_action == TA_SEND_TO_PARALLEL_WORKER)
+ {
+ Assert(winfo);
- oldctx = MemoryContextSwitchTo(ApplyContext);
+ parallel_apply_send_data(winfo, s->len, s->data);
- MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet));
- FileSetInit(MyLogicalRepWorker->stream_fileset);
+ nchanges = 0;
+ elog(DEBUG1, "starting streaming of xid %u", stream_xid);
- MemoryContextSwitchTo(oldctx);
+ /* Cache the parallel apply worker for this transaction. */
+ stream_apply_worker = winfo;
}
+ else if (apply_action == TA_SERIALIZE_TO_FILE)
+ {
+ /*
+ * Notify handle methods we're processing a remote in-progress
+ * transaction.
+ */
+ in_streamed_transaction = true;
- /* open the spool file for this transaction */
- stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+ /*
+ * Since no parallel apply worker is available for the first
+ * stream start, serialize all the changes of the transaction.
+ *
+ * Start a transaction on stream start, this transaction will be
+ * committed on the stream stop unless it is a tablesync worker in
+ * which case it will be committed after processing all the
+ * messages. We need the transaction for handling the buffile,
+ * used for serializing the streaming data and subxact info.
+ */
+ begin_replication_step();
- /* if this is not the first segment, open existing subxact file */
- if (!first_segment)
- subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+ /*
+ * Initialize the worker's stream_fileset if we haven't yet. This will
+ * be used for the entire duration of the worker so create it in a
+ * permanent context. We create this on the very first streaming
+ * message from any transaction and then use it for this and other
+ * streaming transactions. Now, we could create a fileset at the start
+ * of the worker as well but then we won't be sure that it will ever
+ * be used.
+ */
+ if (MyLogicalRepWorker->stream_fileset == NULL)
+ {
+ MemoryContext oldctx;
- pgstat_report_activity(STATE_RUNNING, NULL);
+ oldctx = MemoryContextSwitchTo(ApplyContext);
- end_replication_step();
+ MyLogicalRepWorker->stream_fileset = palloc(sizeof(FileSet));
+ FileSetInit(MyLogicalRepWorker->stream_fileset);
+
+ MemoryContextSwitchTo(oldctx);
+ }
+
+ /* Open the spool file for this transaction. */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /* If this is not the first segment, open existing subxact file. */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ end_replication_step();
+ }
+ else
+ Assert(false);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
}
/*
@@ -1215,58 +1455,76 @@ apply_handle_stream_start(StringInfo s)
static void
apply_handle_stream_stop(StringInfo s)
{
- if (!in_streamed_transaction)
- ereport(ERROR,
- (errcode(ERRCODE_PROTOCOL_VIOLATION),
- errmsg_internal("STREAM STOP message without STREAM START")));
+ ParallelApplyWorkerInfo *winfo = NULL;
+ TransactionApplyAction apply_action;
- /*
- * Close the file with serialized changes, and serialize information about
- * subxacts for the toplevel transaction.
- */
- subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
- stream_close_file();
+ apply_action = get_transaction_apply_action(stream_xid, &winfo);
+
+ if (apply_action == TA_APPLY_IN_PARALLEL_WORKER)
+ {
+ elog(DEBUG1, "[Parallel Apply Worker #%u] ended processing streaming chunk, "
+ "waiting on shm_mq_receive", MyParallelShared->worker_id);
- /* We must be in a valid transaction state */
- Assert(IsTransactionState());
+ pgstat_report_activity(STATE_IDLEINTRANSACTION, NULL);
+ }
+ else if (apply_action == TA_SEND_TO_PARALLEL_WORKER)
+ {
+ Assert(winfo);
- /* Commit the per-stream transaction */
- CommitTransactionCommand();
+ parallel_apply_send_data(winfo, s->len, s->data);
- in_streamed_transaction = false;
+ elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed",
+ stream_xid, nchanges);
- /* Reset per-stream context */
- MemoryContextReset(LogicalStreamingContext);
+ stream_apply_worker = NULL;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+ }
+ else if (apply_action == TA_SERIALIZE_TO_FILE)
+ {
+ if (!in_streamed_transaction)
+ ereport(ERROR,
+ (errcode(ERRCODE_PROTOCOL_VIOLATION),
+ errmsg_internal("STREAM STOP message without STREAM START")));
+
+ /*
+ * Close the file with serialized changes, and serialize information
+ * about subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ /* We must be in a valid transaction state */
+ Assert(IsTransactionState());
+
+ /* Commit the per-stream transaction */
+ CommitTransactionCommand();
+
+ /* Reset per-stream context */
+ MemoryContextReset(LogicalStreamingContext);
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+ }
+ else
+ Assert(false);
+
+ in_streamed_transaction = false;
- pgstat_report_activity(STATE_IDLE, NULL);
reset_apply_error_context_info();
}
/*
- * Handle STREAM abort message.
+ * Handle STREAM ABORT message when the transaction was spilled to disk.
*/
static void
-apply_handle_stream_abort(StringInfo s)
+serialize_stream_abort(TransactionId xid, TransactionId subxid)
{
- TransactionId xid;
- TransactionId subxid;
-
- if (in_streamed_transaction)
- ereport(ERROR,
- (errcode(ERRCODE_PROTOCOL_VIOLATION),
- errmsg_internal("STREAM ABORT message without STREAM STOP")));
-
- logicalrep_read_stream_abort(s, &xid, &subxid);
-
/*
* If the two XIDs are the same, it's in fact abort of toplevel xact, so
* just delete the files with serialized info.
*/
if (xid == subxid)
- {
- set_apply_error_context_xact(xid, InvalidXLogRecPtr);
stream_cleanup_files(MyLogicalRepWorker->subid, xid);
- }
else
{
/*
@@ -1290,8 +1548,6 @@ apply_handle_stream_abort(StringInfo s)
bool found = false;
char path[MAXPGPATH];
- set_apply_error_context_xact(subxid, InvalidXLogRecPtr);
-
subidx = -1;
begin_replication_step();
subxact_info_read(MyLogicalRepWorker->subid, xid);
@@ -1316,7 +1572,6 @@ apply_handle_stream_abort(StringInfo s)
cleanup_subxact_info();
end_replication_step();
CommitTransactionCommand();
- reset_apply_error_context_info();
return;
}
@@ -1339,6 +1594,78 @@ apply_handle_stream_abort(StringInfo s)
end_replication_step();
CommitTransactionCommand();
}
+}
+
+/*
+ * Handle STREAM ABORT message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+ LogicalRepStreamAbortData abort_data;
+ bool read_abort_lsn = false;
+ ParallelApplyWorkerInfo *winfo = NULL;
+ TransactionApplyAction apply_action;
+
+ if (in_streamed_transaction)
+ ereport(ERROR,
+ (errcode(ERRCODE_PROTOCOL_VIOLATION),
+ errmsg_internal("STREAM ABORT message without STREAM STOP")));
+
+ /* Check whether the publisher sends abort_lsn and abort_time. */
+ if (am_parallel_apply_worker())
+ read_abort_lsn = MyParallelShared->proto_version >=
+ LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM;
+
+ logicalrep_read_stream_abort(s, &abort_data, read_abort_lsn);
+
+ xid = abort_data.xid;
+ subxid = abort_data.subxid;
+
+ set_apply_error_context_xact(subxid, abort_data.abort_lsn);
+
+ apply_action = get_transaction_apply_action(xid, &winfo);
+
+ if (apply_action == TA_APPLY_IN_PARALLEL_WORKER)
+ parallel_apply_stream_abort(&abort_data, read_abort_lsn);
+ else if (apply_action == TA_SEND_TO_PARALLEL_WORKER)
+ {
+ Assert(winfo);
+
+ if (subxid == xid)
+ parallel_apply_replorigin_reset();
+
+ /* Send STREAM ABORT message to the parallel apply worker. */
+ parallel_apply_send_data(winfo, s->len, s->data);
+
+ /*
+ * After sending the data to the parallel apply worker, wait for that
+ * worker to finish. This is necessary to maintain commit order which
+ * avoids failures due to transaction dependencies and deadlocks.
+ */
+ parallel_apply_wait_for_free(winfo);
+
+ if (subxid == xid)
+ {
+ parallel_apply_replorigin_setup();
+ parallel_apply_free_worker(winfo, xid);
+
+ elog(DEBUG1, "streamed transaction %u was aborted using an parallel apply worker",
+ xid);
+ }
+ }
+ else if (apply_action == TA_SERIALIZE_TO_FILE)
+ {
+ /*
+ * We are in leader apply worker and the transaction has been
+ * serialized to file.
+ */
+ serialize_stream_abort(xid, subxid);
+ }
+ else
+ Assert(false);
reset_apply_error_context_info();
}
@@ -1470,6 +1797,8 @@ apply_handle_stream_commit(StringInfo s)
{
TransactionId xid;
LogicalRepCommitData commit_data;
+ ParallelApplyWorkerInfo *winfo = NULL;
+ TransactionApplyAction apply_action;
if (in_streamed_transaction)
ereport(ERROR,
@@ -1479,14 +1808,69 @@ apply_handle_stream_commit(StringInfo s)
xid = logicalrep_read_stream_commit(s, &commit_data);
set_apply_error_context_xact(xid, commit_data.commit_lsn);
- elog(DEBUG1, "received commit for streamed transaction %u", xid);
+ apply_action = get_transaction_apply_action(xid, &winfo);
- apply_spooled_messages(xid, commit_data.commit_lsn);
+ if (apply_action == TA_APPLY_IN_PARALLEL_WORKER)
+ {
+ parallel_apply_replorigin_setup();
- apply_handle_commit_internal(&commit_data);
+ apply_handle_commit_internal(&commit_data);
+
+ parallel_apply_replorigin_reset();
+
+ list_free(subxactlist);
+ subxactlist = NIL;
+
+ parallel_apply_set_idle();
+ }
+ else if (apply_action == TA_SEND_TO_PARALLEL_WORKER)
+ {
+ Assert(winfo);
+
+ parallel_apply_replorigin_reset();
+
+ /* Send STREAM COMMIT message to the parallel apply worker. */
+ parallel_apply_send_data(winfo, s->len, s->data);
- /* unlink the files with serialized changes and subxact info */
- stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+ /*
+ * After sending the data to the parallel apply worker, wait for
+ * that worker to finish. This is necessary to maintain commit
+ * order which avoids failures due to transaction dependencies and
+ * deadlocks.
+ */
+ parallel_apply_wait_for_free(winfo);
+ parallel_apply_replorigin_setup();
+
+ pgstat_report_stat(false);
+ store_flush_position(commit_data.end_lsn);
+ stop_skipping_changes();
+
+ parallel_apply_free_worker(winfo, xid);
+
+ elog(DEBUG1, "streamed transaction %u was committed using an parallel apply worker",
+ xid);
+
+ /*
+ * The transaction is either non-empty or skipped, so we clear the
+ * subskiplsn.
+ */
+ clear_subscription_skip_lsn(commit_data.commit_lsn);
+ }
+ else if (apply_action == TA_SERIALIZE_TO_FILE)
+ {
+ /*
+ * The transaction has been serialized to file, so replay all the
+ * spooled operations.
+ */
+ apply_spooled_messages(xid, commit_data.commit_lsn);
+
+ apply_handle_commit_internal(&commit_data);
+
+ /* Unlink the files with serialized changes and subxact info. */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+ }
+ else
+ Assert(false);
/* Process any tables that are being synchronized in parallel. */
process_syncing_tables(commit_data.end_lsn);
@@ -1530,6 +1914,13 @@ apply_handle_commit_internal(LogicalRepCommitData *commit_data)
replorigin_session_origin_timestamp = commit_data->committime;
CommitTransactionCommand();
+
+ if (IsTransactionBlock())
+ {
+ EndTransactionBlock(false);
+ CommitTransactionCommand();
+ }
+
pgstat_report_stat(false);
store_flush_position(commit_data->end_lsn);
@@ -2467,7 +2858,7 @@ apply_handle_truncate(StringInfo s)
/*
* Logical replication protocol message dispatcher.
*/
-static void
+void
apply_dispatch(StringInfo s)
{
LogicalRepMsgType action = pq_getmsgbyte(s);
@@ -2636,6 +3027,10 @@ store_flush_position(XLogRecPtr remote_lsn)
{
FlushPosition *flushpos;
+ /* Skip if not the leader apply worker */
+ if (am_parallel_apply_worker())
+ return;
+
/* Need to do this in permanent context */
MemoryContextSwitchTo(ApplyContext);
@@ -2650,7 +3045,7 @@ store_flush_position(XLogRecPtr remote_lsn)
/* Update statistics of the worker. */
-static void
+void
UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
{
MyLogicalRepWorker->last_lsn = last_lsn;
@@ -2700,6 +3095,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
errcallback.callback = apply_error_callback;
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
+ apply_error_context_stack = error_context_stack;
/* This outer loop iterates once per wait. */
for (;;)
@@ -2914,6 +3310,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
/* Pop the error context stack */
error_context_stack = errcallback.previous;
+ apply_error_context_stack = error_context_stack;
/* All done */
walrcv_endstreaming(LogRepWorkerWalRcvConn, &tli);
@@ -3114,7 +3511,7 @@ maybe_reread_subscription(void)
/*
* Callback from subscription syscache invalidation.
*/
-static void
+void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
@@ -3569,37 +3966,14 @@ start_apply(XLogRecPtr origin_startpos)
PG_END_TRY();
}
-/* Logical Replication Apply worker entry point */
+/*
+ * Initialize the database connection, in-memory subscription and necessary
+ * config options.
+ */
void
-ApplyWorkerMain(Datum main_arg)
+InitializeApplyWorker(void)
{
- int worker_slot = DatumGetInt32(main_arg);
MemoryContext oldctx;
- char originname[NAMEDATALEN];
- XLogRecPtr origin_startpos = InvalidXLogRecPtr;
- char *myslotname = NULL;
- WalRcvStreamOptions options;
- int server_version;
-
- /* Attach to slot */
- logicalrep_worker_attach(worker_slot);
-
- /* Setup signal handling */
- pqsignal(SIGHUP, SignalHandlerForConfigReload);
- pqsignal(SIGTERM, die);
- BackgroundWorkerUnblockSignals();
-
- /*
- * We don't currently need any ResourceOwner in a walreceiver process, but
- * if we did, we could call CreateAuxProcessResourceOwner here.
- */
-
- /* Initialise stats to a sanish value */
- MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
- MyLogicalRepWorker->reply_time = GetCurrentTimestamp();
-
- /* Load the libpq-specific functions */
- load_file("libpqwalreceiver", false);
/* Run as replica session replication role. */
SetConfigOption("session_replication_role", "replica",
@@ -3659,12 +4033,50 @@ ApplyWorkerMain(Datum main_arg)
ereport(LOG,
(errmsg("logical replication table synchronization worker for subscription \"%s\", table \"%s\" has started",
MySubscription->name, get_rel_name(MyLogicalRepWorker->relid))));
+ else if (am_parallel_apply_worker())
+ ereport(LOG,
+ (errmsg("logical replication parallel apply worker for subscription \"%s\" has started",
+ MySubscription->name)));
else
ereport(LOG,
(errmsg("logical replication apply worker for subscription \"%s\" has started",
MySubscription->name)));
CommitTransactionCommand();
+}
+
+/* Logical Replication Apply worker entry point */
+void
+ApplyWorkerMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+ char originname[NAMEDATALEN];
+ XLogRecPtr origin_startpos = InvalidXLogRecPtr;
+ char *myslotname = NULL;
+ WalRcvStreamOptions options;
+ int server_version;
+
+ /* Attach to slot */
+ logicalrep_worker_attach(worker_slot);
+
+ /* Setup signal handling */
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /*
+ * We don't currently need any ResourceOwner in a walreceiver process, but
+ * if we did, we could call CreateAuxProcessResourceOwner here.
+ */
+
+ /* Initialise stats to a sanish value */
+ MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
+ MyLogicalRepWorker->reply_time = GetCurrentTimestamp();
+
+ /* Load the libpq-specific functions */
+ load_file("libpqwalreceiver", false);
+
+ InitializeApplyWorker();
/* Connect to the origin and start the replication. */
elog(DEBUG1, "connecting to publisher using connection string \"%s\"",
@@ -3687,7 +4099,7 @@ ApplyWorkerMain(Datum main_arg)
}
else
{
- /* This is main apply worker */
+ /* This is leader apply worker */
RepOriginId originid;
TimeLineID startpointTLI;
char *err;
@@ -3751,13 +4163,14 @@ ApplyWorkerMain(Datum main_arg)
server_version = walrcv_server_version(LogRepWorkerWalRcvConn);
options.proto.logical.proto_version =
+ server_version >= 160000 ? LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM :
server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM :
server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM :
LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.binary = MySubscription->binary;
- options.proto.logical.streaming = MySubscription->stream;
+ options.proto.logical.streaming = (MySubscription->stream != SUBSTREAM_OFF);
options.proto.logical.twophase = false;
options.proto.logical.origin = pstrdup(MySubscription->origin);
@@ -3855,6 +4268,15 @@ IsLogicalWorker(void)
}
/*
+ * Is current process a logical replication parallel apply worker?
+ */
+bool
+IsLogicalParallelApplyWorker(void)
+{
+ return am_parallel_apply_worker();
+}
+
+/*
* Start skipping changes of the transaction if the given LSN matches the
* LSN specified by subscription's skiplsn.
*/
@@ -3916,7 +4338,8 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn)
XLogRecPtr myskiplsn = MySubscription->skiplsn;
bool started_tx = false;
- if (likely(XLogRecPtrIsInvalid(myskiplsn)))
+ if (likely(XLogRecPtrIsInvalid(myskiplsn)) ||
+ am_parallel_apply_worker())
return;
if (!IsTransactionState())
@@ -3988,7 +4411,7 @@ clear_subscription_skip_lsn(XLogRecPtr finish_lsn)
}
/* Error callback to give more context info about the change being applied */
-static void
+void
apply_error_callback(void *arg)
{
ApplyErrorCallbackArg *errarg = &apply_error_callback_arg;
@@ -4016,23 +4439,47 @@ apply_error_callback(void *arg)
errarg->remote_xid,
LSN_FORMAT_ARGS(errarg->finish_lsn));
}
- else if (errarg->remote_attnum < 0)
- errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u finished at %X/%X",
- errarg->origin_name,
- logicalrep_message_type(errarg->command),
- errarg->rel->remoterel.nspname,
- errarg->rel->remoterel.relname,
- errarg->remote_xid,
- LSN_FORMAT_ARGS(errarg->finish_lsn));
else
- errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u finished at %X/%X",
- errarg->origin_name,
- logicalrep_message_type(errarg->command),
- errarg->rel->remoterel.nspname,
- errarg->rel->remoterel.relname,
- errarg->rel->remoterel.attnames[errarg->remote_attnum],
- errarg->remote_xid,
- LSN_FORMAT_ARGS(errarg->finish_lsn));
+ {
+ if (errarg->remote_attnum < 0)
+ {
+ if (XLogRecPtrIsInvalid(errarg->finish_lsn))
+ errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u",
+ errarg->origin_name,
+ logicalrep_message_type(errarg->command),
+ errarg->rel->remoterel.nspname,
+ errarg->rel->remoterel.relname,
+ errarg->remote_xid);
+ else
+ errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" in transaction %u finished at %X/%X",
+ errarg->origin_name,
+ logicalrep_message_type(errarg->command),
+ errarg->rel->remoterel.nspname,
+ errarg->rel->remoterel.relname,
+ errarg->remote_xid,
+ LSN_FORMAT_ARGS(errarg->finish_lsn));
+ }
+ else
+ {
+ if (XLogRecPtrIsInvalid(errarg->finish_lsn))
+ errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u",
+ errarg->origin_name,
+ logicalrep_message_type(errarg->command),
+ errarg->rel->remoterel.nspname,
+ errarg->rel->remoterel.relname,
+ errarg->rel->remoterel.attnames[errarg->remote_attnum],
+ errarg->remote_xid);
+ else
+ errcontext("processing remote data for replication origin \"%s\" during \"%s\" for replication target relation \"%s.%s\" column \"%s\" in transaction %u finished at %X/%X",
+ errarg->origin_name,
+ logicalrep_message_type(errarg->command),
+ errarg->rel->remoterel.nspname,
+ errarg->rel->remoterel.relname,
+ errarg->rel->remoterel.attnames[errarg->remote_attnum],
+ errarg->remote_xid,
+ LSN_FORMAT_ARGS(errarg->finish_lsn));
+ }
+ }
}
/* Set transaction information of apply error callback */
@@ -4052,3 +4499,26 @@ reset_apply_error_context_info(void)
apply_error_callback_arg.remote_attnum = -1;
set_apply_error_context_xact(InvalidTransactionId, InvalidXLogRecPtr);
}
+
+/*
+ * Return the action to take for the given transaction. Also return the
+ * parallel apply worker information if the action is
+ * TA_SEND_TO_PARALLEL_WORKER.
+ */
+static TransactionApplyAction
+get_transaction_apply_action(TransactionId xid, ParallelApplyWorkerInfo **winfo)
+{
+ if (am_parallel_apply_worker())
+ return TA_APPLY_IN_PARALLEL_WORKER;
+ else if (in_remote_transaction)
+ return TA_APPLY_IN_LEADER_WORKER;
+
+ /*
+ * Check if we are processing this transaction using an parallel apply
+ * worker and if so, send the changes to that worker.
+ */
+ else if ((*winfo = parallel_apply_find_worker(xid)))
+ return TA_SEND_TO_PARALLEL_WORKER;
+ else
+ return TA_SERIALIZE_TO_FILE;
+}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 62e0ffe..d3f19b8 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1843,6 +1843,9 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
XLogRecPtr abort_lsn)
{
ReorderBufferTXN *toptxn;
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+ bool write_abort_lsn = (data->protocol_version >=
+ LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM);
/*
* The abort should happen outside streaming block, even for streamed
@@ -1856,7 +1859,8 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
Assert(rbtxn_is_streamed(toptxn));
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn, abort_lsn,
+ write_abort_lsn);
OutputPluginWrite(ctx, true);
cleanup_rel_sync_cache(toptxn->xid, false);
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 21a9fc0..9cd5428 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -22,6 +22,7 @@
#include "commands/async.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "replication/logicalworker.h"
#include "replication/walsender.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
@@ -657,6 +658,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
+ HandleParallelApplyMessageInterrupt();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 7bec4e4..fb7bd98 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3377,6 +3377,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ if (ParallelApplyMessagePending)
+ HandleParallelApplyMessages();
}
/*
diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c
index 92f24a6..c96647c 100644
--- a/src/backend/utils/activity/wait_event.c
+++ b/src/backend/utils/activity/wait_event.c
@@ -388,6 +388,9 @@ pgstat_get_wait_ipc(WaitEventIPC w)
case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT:
event_name = "HashGrowBucketsReinsert";
break;
+ case WAIT_EVENT_LOGICAL_PARALLEL_APPLY_STATE_CHANGE:
+ event_name = "LogicalParallelApplyStateChange";
+ break;
case WAIT_EVENT_LOGICAL_SYNC_DATA:
event_name = "LogicalSyncData";
break;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 9fbbfb1..faf95bf 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -3223,6 +3223,18 @@ static struct config_int ConfigureNamesInt[] =
},
{
+ {"max_parallel_apply_workers_per_subscription",
+ PGC_SIGHUP,
+ REPLICATION_SUBSCRIBERS,
+ gettext_noop("Maximum number of parallel apply workers per subscription."),
+ NULL,
+ },
+ &max_parallel_apply_workers_per_subscription,
+ 2, 0, MAX_BACKENDS,
+ NULL, NULL, NULL
+ },
+
+ {
{"log_rotation_age", PGC_SIGHUP, LOGGING_WHERE,
gettext_noop("Sets the amount of time to wait before forcing "
"log file rotation."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 90bec05..0f534f5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -360,6 +360,7 @@
#max_logical_replication_workers = 4 # taken from max_worker_processes
# (change requires restart)
#max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers
+#max_parallel_apply_workers_per_subscription = 2 # taken from max_logical_replication_workers
#------------------------------------------------------------------------------
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index d25709a..13c8e10 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4480,7 +4480,7 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 140000)
appendPQExpBufferStr(query, " s.substream,\n");
else
- appendPQExpBufferStr(query, " false AS substream,\n");
+ appendPQExpBufferStr(query, " 'f' AS substream,\n");
if (fout->remoteVersion >= 150000)
appendPQExpBufferStr(query,
@@ -4617,8 +4617,10 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subbinary, "t") == 0)
appendPQExpBufferStr(query, ", binary = true");
- if (strcmp(subinfo->substream, "f") != 0)
+ if (strcmp(subinfo->substream, "t") == 0)
appendPQExpBufferStr(query, ", streaming = on");
+ else if (strcmp(subinfo->substream, "p") == 0)
+ appendPQExpBufferStr(query, ", streaming = parallel");
if (strcmp(subinfo->subtwophasestate, two_phase_disabled) != 0)
appendPQExpBufferStr(query, ", two_phase = on");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 7b98714..79ff745 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -80,7 +80,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subbinary; /* True if the subscription wants the
* publisher to send data in binary */
- bool substream; /* Stream in-progress transactions. */
+ char substream; /* Stream in-progress transactions.
+ * See SUBSTREAM_xxx constants. */
char subtwophasestate; /* Stream two-phase transactions */
@@ -124,7 +125,8 @@ typedef struct Subscription
bool enabled; /* Indicates if the subscription is enabled */
bool binary; /* Indicates if the subscription wants data in
* binary format */
- bool stream; /* Allow streaming in-progress transactions. */
+ char stream; /* Allow streaming in-progress transactions.
+ * See SUBSTREAM_xxx constants. */
char twophasestate; /* Allow streaming two-phase transactions */
bool disableonerr; /* Indicates if the subscription should be
* automatically disabled if a worker error
@@ -137,6 +139,21 @@ typedef struct Subscription
* specified origin */
} Subscription;
+/* Disallow streaming in-progress transactions. */
+#define SUBSTREAM_OFF 'f'
+
+/*
+ * Streaming in-progress transactions are written to a temporary file and
+ * applied only after the transaction is committed on upstream.
+ */
+#define SUBSTREAM_ON 't'
+
+/*
+ * Streaming in-progress transactions are applied immediately via an parallel
+ * apply worker.
+ */
+#define SUBSTREAM_PARALLEL 'p'
+
extern Subscription *GetSubscription(Oid subid, bool missing_ok);
extern void FreeSubscription(Subscription *sub);
extern void DisableSubscription(Oid subid);
diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h
index a6e04d0..c4253a7 100644
--- a/src/include/nodes/pg_list.h
+++ b/src/include/nodes/pg_list.h
@@ -322,6 +322,17 @@ list_nth_oid(const List *list, int n)
return lfirst_oid(list_nth_cell(list, n));
}
+/*
+ * Return the TransactionId value contained in the n'th element of the
+ * specified list.
+ */
+static inline TransactionId
+list_nth_xid(const List *list, int n)
+{
+ Assert(IsA(list, XidList));
+ return lfirst_xid(list_nth_cell(list, n));
+}
+
#define list_nth_node(type,list,n) castNode(type, list_nth(list, n))
/*
diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h
index f1e2821..d513ef5 100644
--- a/src/include/replication/logicallauncher.h
+++ b/src/include/replication/logicallauncher.h
@@ -14,6 +14,7 @@
extern PGDLLIMPORT int max_logical_replication_workers;
extern PGDLLIMPORT int max_sync_workers_per_subscription;
+extern PGDLLIMPORT int max_parallel_apply_workers_per_subscription;
extern void ApplyLauncherRegister(void);
extern void ApplyLauncherMain(Datum main_arg);
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index a771ab8..64adf29 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -32,12 +32,17 @@
*
* LOGICALREP_PROTO_TWOPHASE_VERSION_NUM is the minimum protocol version with
* support for two-phase commit decoding (at prepare time). Introduced in PG15.
+ *
+ * LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM is the minimum protocol version
+ * with support for streaming large transactions using parallel apply
+ * workers. Introduced in PG16.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
#define LOGICALREP_PROTO_VERSION_NUM 1
#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
#define LOGICALREP_PROTO_TWOPHASE_VERSION_NUM 3
-#define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_TWOPHASE_VERSION_NUM
+#define LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM 4
+#define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM
/*
* Logical message types
@@ -175,6 +180,17 @@ typedef struct LogicalRepRollbackPreparedTxnData
char gid[GIDSIZE];
} LogicalRepRollbackPreparedTxnData;
+/*
+ * Transaction protocol information for stream abort.
+ */
+typedef struct LogicalRepStreamAbortData
+{
+ TransactionId xid;
+ TransactionId subxid;
+ XLogRecPtr abort_lsn;
+ TimestampTz abort_time;
+} LogicalRepStreamAbortData;
+
extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn);
extern void logicalrep_read_begin(StringInfo in,
LogicalRepBeginData *begin_data);
@@ -246,9 +262,12 @@ extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn
extern TransactionId logicalrep_read_stream_commit(StringInfo out,
LogicalRepCommitData *commit_data);
extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
- TransactionId subxid);
-extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
- TransactionId *subxid);
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn,
+ bool write_abort_lsn);
+extern void logicalrep_read_stream_abort(StringInfo in,
+ LogicalRepStreamAbortData *abort_data,
+ bool read_abort_lsn);
extern char *logicalrep_message_type(LogicalRepMsgType action);
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index cd1b6e8..96d7e81 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -12,8 +12,14 @@
#ifndef LOGICALWORKER_H
#define LOGICALWORKER_H
+extern PGDLLIMPORT volatile bool ParallelApplyMessagePending;
+
extern void ApplyWorkerMain(Datum main_arg);
+extern void ParallelApplyWorkerMain(Datum main_arg);
extern bool IsLogicalWorker(void);
+extern bool IsLogicalParallelApplyWorker(void);
+extern void HandleParallelApplyMessageInterrupt(void);
+extern void HandleParallelApplyMessages(void);
#endif /* LOGICALWORKER_H */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 8695901..a74b616 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -301,6 +301,7 @@ typedef struct ReorderBufferTXN
{
TimestampTz commit_time;
TimestampTz prepare_time;
+ TimestampTz abort_time;
} xact_time;
/*
@@ -658,9 +659,11 @@ extern void ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid,
extern void ReorderBufferAssignChild(ReorderBuffer *, TransactionId, TransactionId, XLogRecPtr commit_lsn);
extern void ReorderBufferCommitChild(ReorderBuffer *, TransactionId, TransactionId,
XLogRecPtr commit_lsn, XLogRecPtr end_lsn);
-extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn);
+extern void ReorderBufferAbort(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ TimestampTz abort_time);
extern void ReorderBufferAbortOld(ReorderBuffer *, TransactionId xid);
-extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn);
+extern void ReorderBufferForget(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ TimestampTz abort_time);
extern void ReorderBufferInvalidate(ReorderBuffer *, TransactionId, XLogRecPtr lsn);
extern void ReorderBufferSetBaseSnapshot(ReorderBuffer *, TransactionId, XLogRecPtr lsn, struct SnapshotData *snap);
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index 901845a..fe8427b 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -17,8 +17,12 @@
#include "access/xlogdefs.h"
#include "catalog/pg_subscription.h"
#include "datatype/timestamp.h"
+#include "miscadmin.h"
+#include "replication/logicalrelation.h"
#include "storage/fileset.h"
#include "storage/lock.h"
+#include "storage/shm_mq.h"
+#include "storage/shm_toc.h"
#include "storage/spin.h"
@@ -60,6 +64,12 @@ typedef struct LogicalRepWorker
*/
FileSet *stream_fileset;
+ /*
+ * PID of leader apply worker if this slot is used for an parallel apply
+ * worker, InvalidPid otherwise.
+ */
+ pid_t apply_leader_pid;
+
/* Stats. */
XLogRecPtr last_lsn;
TimestampTz last_send_time;
@@ -68,8 +78,74 @@ typedef struct LogicalRepWorker
TimestampTz reply_time;
} LogicalRepWorker;
+/* Struct for saving and restoring apply errcontext information */
+typedef struct ApplyErrorCallbackArg
+{
+ LogicalRepMsgType command; /* 0 if invalid */
+ LogicalRepRelMapEntry *rel;
+
+ /* Remote node information */
+ int remote_attnum; /* -1 if invalid */
+ TransactionId remote_xid;
+ XLogRecPtr finish_lsn;
+ char *origin_name;
+} ApplyErrorCallbackArg;
+
+/*
+ * Struct for sharing information between apply leader apply worker and
+ * parallel apply workers.
+ */
+typedef struct ParallelApplyWorkerShared
+{
+ slock_t mutex;
+
+ bool in_use;
+
+ /* Logical protocol version. */
+ uint32 proto_version;
+
+ /* ID of parallel apply worker */
+ uint32 worker_id;
+
+ /* Information from the corresponding LogicalRepWorker slot */
+ uint16 logicalrep_worker_generation;
+
+ int logicalrep_worker_slot_no;
+} ParallelApplyWorkerShared;
+
+/*
+ * Information which is used to manage the parallel apply worker.
+ */
+typedef struct ParallelApplyWorkerInfo
+{
+ shm_mq_handle *mq_handle;
+
+ /*
+ * The queue used to transfer messages from the parallel apply worker to
+ * the leader apply worker. NULL if the parallel apply worker exited
+ * cleanly.
+ */
+ shm_mq_handle *error_mq_handle;
+
+ dsm_segment *dsm_seg;
+
+ bool in_remote_transaction;
+
+ ParallelApplyWorkerShared volatile *shared;
+} ParallelApplyWorkerInfo;
+
/* Main memory context for apply worker. Permanent during worker lifetime. */
extern PGDLLIMPORT MemoryContext ApplyContext;
+extern PGDLLIMPORT MemoryContext ApplyMessageContext;
+
+extern PGDLLIMPORT ErrorContextCallback *apply_error_context_stack;
+extern PGDLLIMPORT ApplyErrorCallbackArg apply_error_callback_arg;
+
+extern PGDLLIMPORT bool MySubscriptionValid;
+
+extern PGDLLIMPORT volatile ParallelApplyWorkerShared *MyParallelShared;
+
+extern PGDLLIMPORT List *subxactlist;
/* libpqreceiver connection */
extern PGDLLIMPORT struct WalReceiverConn *LogRepWorkerWalRcvConn;
@@ -79,18 +155,24 @@ extern PGDLLIMPORT Subscription *MySubscription;
extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker;
extern PGDLLIMPORT bool in_remote_transaction;
+extern PGDLLIMPORT bool in_streamed_transaction;
+extern PGDLLIMPORT TransactionId stream_xid;
+extern PGDLLIMPORT ParallelApplyWorkerInfo *stream_apply_worker;
extern void logicalrep_worker_attach(int slot);
extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid,
bool only_running);
extern List *logicalrep_workers_find(Oid subid, bool only_running);
-extern void logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname,
- Oid userid, Oid relid);
+extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname,
+ Oid userid, Oid relid,
+ dsm_handle subworker_dsm);
extern void logicalrep_worker_stop(Oid subid, Oid relid);
+extern void logicalrep_worker_stop_by_slot(int slot_no, uint16 generation);
extern void logicalrep_worker_wakeup(Oid subid, Oid relid);
extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker);
extern int logicalrep_sync_worker_count(Oid subid);
+extern int logicalrep_parallel_apply_worker_count(Oid subid);
extern void ReplicationOriginNameForTablesync(Oid suboid, Oid relid,
char *originname, int szorgname);
@@ -103,10 +185,48 @@ extern void process_syncing_tables(XLogRecPtr current_lsn);
extern void invalidate_syncing_table_states(Datum arg, int cacheid,
uint32 hashvalue);
+extern void UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time,
+ bool reply);
+
+extern void apply_dispatch(StringInfo s);
+
+extern void InitializeApplyWorker(void);
+
+/* Function for apply error callback */
+extern void apply_error_callback(void *arg);
+
+extern void subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue);
+
+/* Parallel apply worker setup and interactions */
+extern void parallel_apply_start_worker(TransactionId xid);
+extern ParallelApplyWorkerInfo *parallel_apply_find_worker(TransactionId xid);
+extern void parallel_apply_set_idle(void);
+extern void parallel_apply_free_worker(ParallelApplyWorkerInfo *winfo,
+ TransactionId xid);
+extern void parallel_apply_wait_for_free(ParallelApplyWorkerInfo *winfo);
+extern void parallel_apply_send_data(ParallelApplyWorkerInfo *winfo, Size nbytes,
+ const void *data);
+
+extern void parallel_apply_subxact_info_add(TransactionId current_xid);
+extern void parallel_apply_stream_abort(LogicalRepStreamAbortData *abort_data,
+ bool read_abort_lsn);
+extern void parallel_apply_savepoint_name(Oid suboid, Oid relid,
+ char *spname, int szsp);
+extern void parallel_apply_replorigin_setup(void);
+extern void parallel_apply_replorigin_reset(void);
+
+#define isParallelApplyWorker(worker) (worker->apply_leader_pid != InvalidPid)
+
static inline bool
am_tablesync_worker(void)
{
return OidIsValid(MyLogicalRepWorker->relid);
}
+static inline bool
+am_parallel_apply_worker(void)
+{
+ return isParallelApplyWorker(MyLogicalRepWorker);
+}
+
#endif /* WORKER_INTERNAL_H */
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index ee63690..528ef73 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h
index 6f2d561..b2cdd55 100644
--- a/src/include/utils/wait_event.h
+++ b/src/include/utils/wait_event.h
@@ -105,6 +105,7 @@ typedef enum
WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATE,
WAIT_EVENT_HASH_GROW_BUCKETS_ELECT,
WAIT_EVENT_HASH_GROW_BUCKETS_REINSERT,
+ WAIT_EVENT_LOGICAL_PARALLEL_APPLY_STATE_CHANGE,
WAIT_EVENT_LOGICAL_SYNC_DATA,
WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE,
WAIT_EVENT_MQ_INTERNAL,
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index ef0ebf9..99b9e86 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -217,9 +217,9 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
(1 row)
DROP SUBSCRIPTION regress_testsub;
--- fail - streaming must be boolean
+-- fail - streaming must be boolean or 'parallel'
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo);
-ERROR: streaming requires a Boolean value
+ERROR: streaming requires a Boolean value or "parallel"
-- now it works
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = true);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
@@ -230,6 +230,14 @@ WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ..
regress_testsub | regress_subscription_user | f | {testpub} | f | t | d | f | any | off | dbname=regress_doesnotexist | 0/0
(1 row)
+ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | p | d | f | any | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 4425faf..ed148e4 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -156,7 +156,7 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
--- fail - streaming must be boolean
+-- fail - streaming must be boolean or 'parallel'
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, streaming = foo);
-- now it works
@@ -164,6 +164,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
\dRs+
+ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
+
+\dRs+
+
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8ad112c..b5a4fde 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1485,6 +1485,7 @@ LogicalRepRelId
LogicalRepRelMapEntry
LogicalRepRelation
LogicalRepRollbackPreparedTxnData
+LogicalRepStreamAbortData
LogicalRepTupleData
LogicalRepTyp
LogicalRepWorker
@@ -1676,6 +1677,9 @@ OverrideStackEntry
OverridingKind
PACE_HEADER
PACL
+ParallelApplyWorkerEntry
+ParallelApplyWorkerInfo
+ParallelApplyWorkerShared
PATH
PBOOL
PCtxtHandle
--
2.7.2.windows.1
[application/octet-stream] v26-0002-Test-streaming-parallel-option-in-tap-test.patch (75.2K, ../OS0PR01MB5716546863340CA19EC5F8C394789@OS0PR01MB5716.jpnprd01.prod.outlook.com/4-v26-0002-Test-streaming-parallel-option-in-tap-test.patch)
download | inline diff:
From 712fb347f1bb6f15ab4913da0e421cca652557b5 Mon Sep 17 00:00:00 2001
From: "shiy.fnst" <[email protected]>
Date: Fri, 13 May 2022 14:50:30 +0800
Subject: [PATCH v26 2/5] Test streaming parallel option in tap test
Change all TAP tests using the SUBSCRIPTION "streaming" parameter, so they
now test both 'on' and 'parallel' values.
---
src/test/subscription/t/015_stream.pl | 233 ++++++---
src/test/subscription/t/016_stream_subxact.pl | 144 ++++--
src/test/subscription/t/017_stream_ddl.pl | 222 ++++++---
.../subscription/t/018_stream_subxact_abort.pl | 234 ++++++---
.../subscription/t/019_stream_subxact_ddl_abort.pl | 135 ++++-
src/test/subscription/t/022_twophase_cascade.pl | 395 +++++++++------
src/test/subscription/t/023_twophase_stream.pl | 541 ++++++++++++---------
7 files changed, 1259 insertions(+), 645 deletions(-)
diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl
index cbaa327..6844cc9 100644
--- a/src/test/subscription/t/015_stream.pl
+++ b/src/test/subscription/t/015_stream.pl
@@ -8,6 +8,150 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
+# Check the log that the streamed transaction was completed successfully
+# reported by parallel apply worker.
+sub check_parallel_log
+{
+ my ($node_subscriber, $offset, $is_parallel, $action)= @_;
+ my $parallel_message = '';
+
+ if ($is_parallel)
+ {
+ $parallel_message = join("", 'streamed transaction \d+ was ', $action, ' using an parallel apply worker');
+
+ $node_subscriber->wait_for_log(qr/$parallel_message/, $offset);
+ }
+}
+
+# Encapsulate all the common test steps which are related to "streaming"
+# parameter so the same code can be run both for the streaming=on and
+# streaming=parallel cases.
+sub test_streaming
+{
+ my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
+
+ # Interleave a pair of transactions, each exceeding the 64kB limit.
+ my $in = '';
+ my $out = '';
+
+ my $offset = 0;
+
+ my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
+
+ my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
+ on_error_stop => 0);
+
+ # If "streaming" parameter is specified as "parallel", we need to check
+ # that streamed transaction was applied using an parallel apply worker.
+ # We have to look for the DEBUG1 log messages about that, so bump up the
+ # log verbosity.
+ if ($is_parallel)
+ {
+ $node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = debug1");
+ $node_subscriber->reload;
+ }
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ $in .= q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+ UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+ DELETE FROM test_tab WHERE mod(a,3) = 0;
+ };
+ $h->pump_nb;
+
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i);
+ DELETE FROM test_tab WHERE a > 5000;
+ COMMIT;
+ });
+
+ $in .= q{
+ COMMIT;
+ \q
+ };
+ $h->finish; # errors make the next test fail, so ignore them here
+
+ $node_publisher->wait_for_catchup($appname);
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
+
+ my $result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+ is($result, qq(3334|3334|3334),
+ 'check extra columns contain local defaults');
+
+ # Test the streaming in binary mode
+ $node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (binary = on)");
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ # Insert, update and delete enough rows to exceed the 64kB limit.
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i);
+ UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+ DELETE FROM test_tab WHERE mod(a,3) = 0;
+ COMMIT;
+ });
+
+ $node_publisher->wait_for_catchup($appname);
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
+
+ $result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+ is($result, qq(6667|6667|6667),
+ 'check extra columns contain local defaults');
+
+ # Change the local values of the extra columns on the subscriber,
+ # update publisher, and check that subscriber retains the expected
+ # values. This is to ensure that non-streaming transactions behave
+ # properly after a streaming transaction.
+ $node_subscriber->safe_psql('postgres',
+ "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'"
+ );
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ $node_publisher->safe_psql('postgres',
+ "UPDATE test_tab SET b = md5(a::text)");
+
+ $node_publisher->wait_for_catchup($appname);
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
+
+ $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab"
+ );
+ is($result, qq(6667|6667|6667),
+ 'check extra columns contain locally changed data');
+
+ # Cleanup the test data
+ $node_publisher->safe_psql('postgres',
+ "DELETE FROM test_tab WHERE (a > 2)");
+ $node_publisher->wait_for_catchup($appname);
+
+ # Reset the log verbosity.
+ if ($is_parallel)
+ {
+ $node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = warning");
+ $node_subscriber->reload;
+ }
+}
+
# Create publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
@@ -37,6 +181,10 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR TABLE test_tab");
my $appname = 'tap_sub';
+
+################################
+# Test using streaming mode 'on'
+################################
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
@@ -49,82 +197,25 @@ my $result =
"SELECT count(*), count(c), count(d = 999) FROM test_tab");
is($result, qq(2|2|2), 'check initial data was copied to subscriber');
-# Interleave a pair of transactions, each exceeding the 64kB limit.
-my $in = '';
-my $out = '';
-
-my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
-my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
- on_error_stop => 0);
-
-$in .= q{
-BEGIN;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
-UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
-DELETE FROM test_tab WHERE mod(a,3) = 0;
-};
-$h->pump_nb;
-
-$node_publisher->safe_psql(
- 'postgres', q{
-BEGIN;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 9999) s(i);
-DELETE FROM test_tab WHERE a > 5000;
-COMMIT;
-});
-
-$in .= q{
-COMMIT;
-\q
-};
-$h->finish; # errors make the next test fail, so ignore them here
-
-$node_publisher->wait_for_catchup($appname);
-
-$result =
- $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(c), count(d = 999) FROM test_tab");
-is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+test_streaming($node_publisher, $node_subscriber, $appname, 0);
-# Test the streaming in binary mode
-$node_subscriber->safe_psql('postgres',
- "ALTER SUBSCRIPTION tap_sub SET (binary = on)");
-
-# Insert, update and delete enough rows to exceed the 64kB limit.
-$node_publisher->safe_psql(
- 'postgres', q{
-BEGIN;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001, 10000) s(i);
-UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
-DELETE FROM test_tab WHERE mod(a,3) = 0;
-COMMIT;
-});
-
-$node_publisher->wait_for_catchup($appname);
-
-$result =
- $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(c), count(d = 999) FROM test_tab");
-is($result, qq(6667|6667|6667), 'check extra columns contain local defaults');
+######################################
+# Test using streaming mode 'parallel'
+######################################
+my $oldpid = $node_publisher->safe_psql('postgres',
+ "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
+);
-# Change the local values of the extra columns on the subscriber,
-# update publisher, and check that subscriber retains the expected
-# values. This is to ensure that non-streaming transactions behave
-# properly after a streaming transaction.
$node_subscriber->safe_psql('postgres',
- "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'"
-);
-$node_publisher->safe_psql('postgres',
- "UPDATE test_tab SET b = md5(a::text)");
+ "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel, binary = off)");
-$node_publisher->wait_for_catchup($appname);
+$node_publisher->poll_query_until('postgres',
+ "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
+ )
+ or die
+ "Timed out while waiting for apply to restart after changing SUBSCRIPTION";
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab"
-);
-is($result, qq(6667|6667|6667),
- 'check extra columns contain locally changed data');
+test_streaming($node_publisher, $node_subscriber, $appname, 1);
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl
index bc0a9cd..c5a10dc 100644
--- a/src/test/subscription/t/016_stream_subxact.pl
+++ b/src/test/subscription/t/016_stream_subxact.pl
@@ -8,6 +8,95 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
+# Check the log that the streamed transaction was completed successfully
+# reported by parallel apply worker.
+sub check_parallel_log
+{
+ my ($node_subscriber, $offset, $is_parallel, $action)= @_;
+ my $parallel_message = '';
+
+ if ($is_parallel)
+ {
+ $parallel_message = join("", 'streamed transaction \d+ was ', $action, ' using an parallel apply worker');
+
+ $node_subscriber->wait_for_log(qr/$parallel_message/, $offset);
+ }
+}
+
+# Encapsulate all the common test steps which are related to "streaming"
+# parameter so the same code can be run both for the streaming=on and
+# streaming=parallel cases.
+sub test_streaming
+{
+ my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
+
+ my $offset = 0;
+
+ # If "streaming" parameter is specified as "parallel", we need to check
+ # that streamed transaction was applied using an parallel apply worker.
+ # We have to look for the DEBUG1 log messages about that, so bump up the
+ # log verbosity.
+ if ($is_parallel)
+ {
+ $node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = debug1");
+ $node_subscriber->reload;
+ }
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ # Insert, update and delete enough rows to exceed 64kB limit.
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+ UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+ DELETE FROM test_tab WHERE mod(a,3) = 0;
+ SAVEPOINT s1;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+ UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+ DELETE FROM test_tab WHERE mod(a,3) = 0;
+ SAVEPOINT s2;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+ UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+ DELETE FROM test_tab WHERE mod(a,3) = 0;
+ SAVEPOINT s3;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+ UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+ DELETE FROM test_tab WHERE mod(a,3) = 0;
+ SAVEPOINT s4;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+ UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+ DELETE FROM test_tab WHERE mod(a,3) = 0;
+ COMMIT;
+ });
+
+ $node_publisher->wait_for_catchup($appname);
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
+
+ my $result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+ is($result, qq(1667|1667|1667),
+ 'check data was copied to subscriber in streaming mode and extra columns contain local defaults'
+ );
+
+ # Cleanup the test data
+ $node_publisher->safe_psql('postgres',
+ "DELETE FROM test_tab WHERE (a > 2)");
+ $node_publisher->wait_for_catchup($appname);
+
+ # Reset the log verbosity.
+ if ($is_parallel)
+ {
+ $node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = warning");
+ $node_subscriber->reload;
+ }
+}
+
# Create publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
@@ -37,6 +126,10 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR TABLE test_tab");
my $appname = 'tap_sub';
+
+################################
+# Test using streaming mode 'on'
+################################
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
@@ -49,41 +142,26 @@ my $result =
"SELECT count(*), count(c), count(d = 999) FROM test_tab");
is($result, qq(2|2|2), 'check initial data was copied to subscriber');
-# Insert, update and delete enough rows to exceed 64kB limit.
-$node_publisher->safe_psql(
- 'postgres', q{
-BEGIN;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
-UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
-DELETE FROM test_tab WHERE mod(a,3) = 0;
-SAVEPOINT s1;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
-UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
-DELETE FROM test_tab WHERE mod(a,3) = 0;
-SAVEPOINT s2;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
-UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
-DELETE FROM test_tab WHERE mod(a,3) = 0;
-SAVEPOINT s3;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
-UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
-DELETE FROM test_tab WHERE mod(a,3) = 0;
-SAVEPOINT s4;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
-UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
-DELETE FROM test_tab WHERE mod(a,3) = 0;
-COMMIT;
-});
-
-$node_publisher->wait_for_catchup($appname);
-
-$result =
- $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(c), count(d = 999) FROM test_tab");
-is($result, qq(1667|1667|1667),
- 'check data was copied to subscriber in streaming mode and extra columns contain local defaults'
+test_streaming($node_publisher, $node_subscriber, $appname, 0);
+
+######################################
+# Test using streaming mode 'parallel'
+######################################
+my $oldpid = $node_publisher->safe_psql('postgres',
+ "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
);
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)");
+
+$node_publisher->poll_query_until('postgres',
+ "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
+ )
+ or die
+ "Timed out while waiting for apply to restart after changing SUBSCRIPTION";
+
+test_streaming($node_publisher, $node_subscriber, $appname, 1);
+
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl
index 866f151..0beca11 100644
--- a/src/test/subscription/t/017_stream_ddl.pl
+++ b/src/test/subscription/t/017_stream_ddl.pl
@@ -8,6 +8,139 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
+# Check the log that the streamed transaction was completed successfully
+# reported by parallel apply worker.
+sub check_parallel_log
+{
+ my ($node_subscriber, $offset, $is_parallel, $action)= @_;
+ my $parallel_message = '';
+
+ if ($is_parallel)
+ {
+ $parallel_message = join("", 'streamed transaction \d+ was ', $action, ' using an parallel apply worker');
+
+ $node_subscriber->wait_for_log(qr/$parallel_message/, $offset);
+ }
+}
+
+# Encapsulate all the common test steps which are related to "streaming"
+# parameter so the same code can be run both for the streaming=on and
+# streaming=parallel cases.
+sub test_streaming
+{
+ my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
+
+ my $offset = 0;
+
+ # a small (non-streamed) transaction with DDL and DML
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab VALUES (3, md5(3::text));
+ ALTER TABLE test_tab ADD COLUMN c INT;
+ SAVEPOINT s1;
+ INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+ COMMIT;
+ });
+
+ # If "streaming" parameter is specified as "parallel", we need to check
+ # that streamed transaction was applied using an parallel apply worker.
+ # We have to look for the DEBUG1 log messages about that, so bump up the
+ # log verbosity.
+ if ($is_parallel)
+ {
+ $node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = debug1");
+ $node_subscriber->reload;
+ }
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ # large (streamed) transaction with DDL and DML
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ ALTER TABLE test_tab ADD COLUMN d INT;
+ SAVEPOINT s1;
+ INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+ COMMIT;
+ });
+
+ # a small (non-streamed) transaction with DDL and DML
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ ALTER TABLE test_tab ADD COLUMN e INT;
+ SAVEPOINT s1;
+ INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+ COMMIT;
+ });
+
+ $node_publisher->wait_for_catchup($appname);
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
+
+ my $result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+ is($result, qq(2002|1999|1002|1),
+ 'check data was copied to subscriber in streaming mode and extra columns contain local defaults'
+ );
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ # A large (streamed) transaction with DDL and DML. One of the DDL is performed
+ # after DML to ensure that we invalidate the schema sent for test_tab so that
+ # the next transaction has to send the schema again.
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i);
+ ALTER TABLE test_tab ADD COLUMN f INT;
+ COMMIT;
+ });
+
+ # A small transaction that won't get streamed. This is just to ensure that we
+ # send the schema again to reflect the last column added in the previous test.
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i);
+ COMMIT;
+ });
+
+ $node_publisher->wait_for_catchup($appname);
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
+
+ $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"
+ );
+ is($result, qq(5005|5002|4005|3004|5),
+ 'check data was copied to subscriber for both streaming and non-streaming transactions'
+ );
+
+ # Cleanup the test data
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ DELETE FROM test_tab WHERE (a > 2);
+ ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f;
+ });
+ $node_publisher->wait_for_catchup($appname);
+
+ # Reset the log verbosity.
+ if ($is_parallel)
+ {
+ $node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = warning");
+ $node_subscriber->reload;
+ }
+}
+
# Create publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
@@ -37,6 +170,10 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR TABLE test_tab");
my $appname = 'tap_sub';
+
+################################
+# Test using streaming mode 'on'
+################################
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
@@ -49,77 +186,26 @@ my $result =
"SELECT count(*), count(c), count(d = 999) FROM test_tab");
is($result, qq(2|0|0), 'check initial data was copied to subscriber');
-# a small (non-streamed) transaction with DDL and DML
-$node_publisher->safe_psql(
- 'postgres', q{
-BEGIN;
-INSERT INTO test_tab VALUES (3, md5(3::text));
-ALTER TABLE test_tab ADD COLUMN c INT;
-SAVEPOINT s1;
-INSERT INTO test_tab VALUES (4, md5(4::text), -4);
-COMMIT;
-});
-
-# large (streamed) transaction with DDL and DML
-$node_publisher->safe_psql(
- 'postgres', q{
-BEGIN;
-INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
-ALTER TABLE test_tab ADD COLUMN d INT;
-SAVEPOINT s1;
-INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
-COMMIT;
-});
-
-# a small (non-streamed) transaction with DDL and DML
-$node_publisher->safe_psql(
- 'postgres', q{
-BEGIN;
-INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
-ALTER TABLE test_tab ADD COLUMN e INT;
-SAVEPOINT s1;
-INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
-COMMIT;
-});
-
-$node_publisher->wait_for_catchup($appname);
-
-$result =
- $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
-is($result, qq(2002|1999|1002|1),
- 'check data was copied to subscriber in streaming mode and extra columns contain local defaults'
-);
+test_streaming($node_publisher, $node_subscriber, $appname, 0);
-# A large (streamed) transaction with DDL and DML. One of the DDL is performed
-# after DML to ensure that we invalidate the schema sent for test_tab so that
-# the next transaction has to send the schema again.
-$node_publisher->safe_psql(
- 'postgres', q{
-BEGIN;
-INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(2003,5000) s(i);
-ALTER TABLE test_tab ADD COLUMN f INT;
-COMMIT;
-});
-
-# A small transaction that won't get streamed. This is just to ensure that we
-# send the schema again to reflect the last column added in the previous test.
-$node_publisher->safe_psql(
- 'postgres', q{
-BEGIN;
-INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i, 4*i FROM generate_series(5001,5005) s(i);
-COMMIT;
-});
-
-$node_publisher->wait_for_catchup($appname);
-
-$result =
- $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab");
-is($result, qq(5005|5002|4005|3004|5),
- 'check data was copied to subscriber for both streaming and non-streaming transactions'
+######################################
+# Test using streaming mode 'parallel'
+######################################
+my $oldpid = $node_publisher->safe_psql('postgres',
+ "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
);
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)");
+
+$node_publisher->poll_query_until('postgres',
+ "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
+ )
+ or die
+ "Timed out while waiting for apply to restart after changing SUBSCRIPTION";
+
+test_streaming($node_publisher, $node_subscriber, $appname, 1);
+
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl
index 551f16d..12f693b 100644
--- a/src/test/subscription/t/018_stream_subxact_abort.pl
+++ b/src/test/subscription/t/018_stream_subxact_abort.pl
@@ -8,6 +8,146 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
+# Check the log that the streamed transaction was completed successfully
+# reported by parallel apply worker.
+sub check_parallel_log
+{
+ my ($node_subscriber, $offset, $is_parallel, $action)= @_;
+ my $parallel_message = '';
+
+ if ($is_parallel)
+ {
+ $parallel_message = join("", 'streamed transaction \d+ was ', $action, ' using an parallel apply worker');
+
+ $node_subscriber->wait_for_log(qr/$parallel_message/, $offset);
+ }
+}
+
+# Encapsulate all the common test steps which are related to "streaming"
+# parameter so the same code can be run both for the streaming=on and
+# streaming=parallel cases.
+sub test_streaming
+{
+ my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
+
+ my $offset = 0;
+
+ # If "streaming" parameter is specified as "parallel", we need to check
+ # that streamed transaction was applied using an parallel apply worker.
+ # We have to look for the DEBUG1 log messages about that, so bump up the
+ # log verbosity.
+ if ($is_parallel)
+ {
+ $node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = debug1");
+ $node_subscriber->reload;
+ }
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ # large (streamed) transaction with DDL, DML and ROLLBACKs
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ SAVEPOINT s1;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+ SAVEPOINT s2;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+ SAVEPOINT s3;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ ROLLBACK TO s2;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ ROLLBACK TO s1;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+ SAVEPOINT s4;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i);
+ SAVEPOINT s5;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i);
+ COMMIT;
+ });
+
+ $node_publisher->wait_for_catchup($appname);
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
+
+ my $result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(c) FROM test_tab");
+ is($result, qq(2000|0),
+ 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults'
+ );
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ # large (streamed) transaction with subscriber receiving out of order
+ # subtransaction ROLLBACKs
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i);
+ SAVEPOINT s1;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i);
+ SAVEPOINT s2;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i);
+ SAVEPOINT s3;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i);
+ RELEASE s2;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i);
+ ROLLBACK TO s1;
+ COMMIT;
+ });
+
+ $node_publisher->wait_for_catchup($appname);
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
+
+ $result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(c) FROM test_tab");
+ is($result, qq(2500|0),
+ 'check rollback to savepoint was reflected on subscriber');
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ # large (streamed) transaction with subscriber receiving rollback
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i);
+ SAVEPOINT s1;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i);
+ SAVEPOINT s2;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i);
+ ROLLBACK;
+ });
+
+ $node_publisher->wait_for_catchup($appname);
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "aborted");
+
+ $result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(c) FROM test_tab");
+ is($result, qq(2500|0), 'check rollback was reflected on subscriber');
+
+ # Cleanup the test data
+ $node_publisher->safe_psql('postgres',
+ "DELETE FROM test_tab WHERE (a > 2)");
+ $node_publisher->wait_for_catchup($appname);
+
+ # Reset the log verbosity.
+ if ($is_parallel)
+ {
+ $node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = warning");
+ $node_subscriber->reload;
+ }
+}
+
# Create publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
@@ -36,6 +176,10 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR TABLE test_tab");
my $appname = 'tap_sub';
+
+################################
+# Test using streaming mode 'on'
+################################
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
@@ -48,81 +192,25 @@ my $result =
"SELECT count(*), count(c) FROM test_tab");
is($result, qq(2|0), 'check initial data was copied to subscriber');
-# large (streamed) transaction with DDL, DML and ROLLBACKs
-$node_publisher->safe_psql(
- 'postgres', q{
-BEGIN;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
-SAVEPOINT s1;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
-SAVEPOINT s2;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
-SAVEPOINT s3;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
-ROLLBACK TO s2;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
-ROLLBACK TO s1;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
-SAVEPOINT s4;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3001,3500) s(i);
-SAVEPOINT s5;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3501,4000) s(i);
-COMMIT;
-});
-
-$node_publisher->wait_for_catchup($appname);
-
-$result =
- $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(c) FROM test_tab");
-is($result, qq(2000|0),
- 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults'
+test_streaming($node_publisher, $node_subscriber, $appname, 0);
+
+######################################
+# Test using streaming mode 'parallel'
+######################################
+my $oldpid = $node_publisher->safe_psql('postgres',
+ "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
);
-# large (streamed) transaction with subscriber receiving out of order
-# subtransaction ROLLBACKs
-$node_publisher->safe_psql(
- 'postgres', q{
-BEGIN;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(4001,4500) s(i);
-SAVEPOINT s1;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(5001,5500) s(i);
-SAVEPOINT s2;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(6001,6500) s(i);
-SAVEPOINT s3;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(7001,7500) s(i);
-RELEASE s2;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8001,8500) s(i);
-ROLLBACK TO s1;
-COMMIT;
-});
-
-$node_publisher->wait_for_catchup($appname);
-
-$result =
- $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(c) FROM test_tab");
-is($result, qq(2500|0),
- 'check rollback to savepoint was reflected on subscriber');
-
-# large (streamed) transaction with subscriber receiving rollback
-$node_publisher->safe_psql(
- 'postgres', q{
-BEGIN;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(8501,9000) s(i);
-SAVEPOINT s1;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9001,9500) s(i);
-SAVEPOINT s2;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(9501,10000) s(i);
-ROLLBACK;
-});
-
-$node_publisher->wait_for_catchup($appname);
-
-$result =
- $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(c) FROM test_tab");
-is($result, qq(2500|0), 'check rollback was reflected on subscriber');
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)");
+
+$node_publisher->poll_query_until('postgres',
+ "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
+ )
+ or die
+ "Timed out while waiting for apply to restart after changing SUBSCRIPTION";
+
+test_streaming($node_publisher, $node_subscriber, $appname, 1);
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl
index 4d7da82..90a77e5 100644
--- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl
@@ -9,6 +9,92 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
+# Check the log that the streamed transaction was completed successfully
+# reported by parallel apply worker.
+sub check_parallel_log
+{
+ my ($node_subscriber, $offset, $is_parallel, $action)= @_;
+ my $parallel_message = '';
+
+ if ($is_parallel)
+ {
+ $parallel_message = join("", 'streamed transaction \d+ was ', $action, ' using an parallel apply worker');
+
+ $node_subscriber->wait_for_log(qr/$parallel_message/, $offset);
+ }
+}
+
+# Encapsulate all the common test steps which are related to "streaming"
+# parameter so the same code can be run both for the streaming=on and
+# streaming=parallel cases.
+sub test_streaming
+{
+ my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
+
+ my $offset = 0;
+
+ # If "streaming" parameter is specified as "parallel", we need to check
+ # that streamed transaction was applied using an parallel apply worker.
+ # We have to look for the DEBUG1 log messages about that, so bump up the
+ # log verbosity.
+ if ($is_parallel)
+ {
+ $node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = debug1");
+ $node_subscriber->reload;
+ }
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ # large (streamed) transaction with DDL, DML and ROLLBACKs
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ ALTER TABLE test_tab ADD COLUMN c INT;
+ SAVEPOINT s1;
+ INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ ALTER TABLE test_tab ADD COLUMN d INT;
+ SAVEPOINT s2;
+ INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ ALTER TABLE test_tab ADD COLUMN e INT;
+ SAVEPOINT s3;
+ INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ ALTER TABLE test_tab DROP COLUMN c;
+ ROLLBACK TO s1;
+ INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+ COMMIT;
+ });
+
+ $node_publisher->wait_for_catchup($appname);
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
+
+ my $result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(c) FROM test_tab");
+ is($result, qq(1000|500),
+ 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults'
+ );
+
+ # Cleanup the test data
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ DELETE FROM test_tab WHERE (a > 2);
+ ALTER TABLE test_tab DROP COLUMN c;
+ });
+ $node_publisher->wait_for_catchup($appname);
+
+ # Reset the log verbosity.
+ if ($is_parallel)
+ {
+ $node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = warning");
+ $node_subscriber->reload;
+ }
+}
+
# Create publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
@@ -37,6 +123,10 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR TABLE test_tab");
my $appname = 'tap_sub';
+
+################################
+# Test using streaming mode 'on'
+################################
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
@@ -49,35 +139,26 @@ my $result =
"SELECT count(*), count(c) FROM test_tab");
is($result, qq(2|0), 'check initial data was copied to subscriber');
-# large (streamed) transaction with DDL, DML and ROLLBACKs
-$node_publisher->safe_psql(
- 'postgres', q{
-BEGIN;
-INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
-ALTER TABLE test_tab ADD COLUMN c INT;
-SAVEPOINT s1;
-INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
-ALTER TABLE test_tab ADD COLUMN d INT;
-SAVEPOINT s2;
-INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
-ALTER TABLE test_tab ADD COLUMN e INT;
-SAVEPOINT s3;
-INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
-ALTER TABLE test_tab DROP COLUMN c;
-ROLLBACK TO s1;
-INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
-COMMIT;
-});
-
-$node_publisher->wait_for_catchup($appname);
-
-$result =
- $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(c) FROM test_tab");
-is($result, qq(1000|500),
- 'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults'
+test_streaming($node_publisher, $node_subscriber, $appname, 0);
+
+######################################
+# Test using streaming mode 'parallel'
+######################################
+my $oldpid = $node_publisher->safe_psql('postgres',
+ "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
);
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)");
+
+$node_publisher->poll_query_until('postgres',
+ "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
+ )
+ or die
+ "Timed out while waiting for apply to restart after changing SUBSCRIPTION";
+
+test_streaming($node_publisher, $node_subscriber, $appname, 1);
+
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl
index 7a797f3..970fd85 100644
--- a/src/test/subscription/t/022_twophase_cascade.pl
+++ b/src/test/subscription/t/022_twophase_cascade.pl
@@ -11,6 +11,240 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
+# Check the log that the streamed transaction was completed successfully
+# reported by parallel apply worker.
+sub check_parallel_log
+{
+ my ($node_subscriber, $offset, $streaming_mode, $action)= @_;
+ my $parallel_message = '';
+
+ if ($streaming_mode eq 'parallel')
+ {
+ $parallel_message = join("", 'streamed transaction \d+ was ', $action, ' using an parallel apply worker');
+
+ $node_subscriber->wait_for_log(qr/$parallel_message/, $offset);
+ }
+}
+
+# Encapsulate all the common test steps which are related to "streaming" parameter
+# so the same code can be run both for the streaming=on and streaming=parallel
+# cases.
+sub test_streaming
+{
+ my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) =
+ @_;
+
+ my $offset_B = 0;
+ my $offset_C = 0;
+
+ my $oldpid_B = $node_A->safe_psql(
+ 'postgres', "
+ SELECT pid FROM pg_stat_replication
+ WHERE application_name = '$appname_B' AND state = 'streaming';");
+ my $oldpid_C = $node_B->safe_psql(
+ 'postgres', "
+ SELECT pid FROM pg_stat_replication
+ WHERE application_name = '$appname_C' AND state = 'streaming';");
+
+ # Setup logical replication streaming mode
+
+ $node_B->safe_psql(
+ 'postgres', "
+ ALTER SUBSCRIPTION tap_sub_B
+ SET (streaming = $streaming_mode);");
+ $node_C->safe_psql(
+ 'postgres', "
+ ALTER SUBSCRIPTION tap_sub_C
+ SET (streaming = $streaming_mode)");
+
+ # Wait for subscribers to finish initialization
+
+ $node_A->poll_query_until(
+ 'postgres', "
+ SELECT pid != $oldpid_B FROM pg_stat_replication
+ WHERE application_name = '$appname_B' AND state = 'streaming';"
+ ) or die "Timed out while waiting for apply to restart";
+ $node_B->poll_query_until(
+ 'postgres', "
+ SELECT pid != $oldpid_C FROM pg_stat_replication
+ WHERE application_name = '$appname_C' AND state = 'streaming';"
+ ) or die "Timed out while waiting for apply to restart";
+
+ ###############################
+ # Test 2PC PREPARE / COMMIT PREPARED.
+ # 1. Data is streamed as a 2PC transaction.
+ # 2. Then do commit prepared.
+ #
+ # Expect all data is replicated on subscriber(s) after the commit.
+ ###############################
+
+ # If "streaming" parameter is specified as "parallel", we need to check
+ # that streamed transaction was prepared using an parallel apply worker.
+ # We have to look for the DEBUG1 log messages about that, so bump up the
+ # log verbosity.
+ if ($streaming_mode eq 'parallel')
+ {
+ $node_B->append_conf('postgresql.conf', "log_min_messages = debug1");
+ $node_B->reload;
+
+ $node_C->append_conf('postgresql.conf', "log_min_messages = debug1");
+ $node_C->reload;
+ }
+
+ # Check the subscriber log from now on.
+ $offset_B = -s $node_B->logfile;
+ $offset_C = -s $node_C->logfile;
+
+ # Insert, update and delete enough rows to exceed the 64kB limit.
+ # Then 2PC PREPARE
+ $node_A->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+ UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+ DELETE FROM test_tab WHERE mod(a,3) = 0;
+ PREPARE TRANSACTION 'test_prepared_tab';});
+
+ $node_A->wait_for_catchup($appname_B);
+ $node_B->wait_for_catchup($appname_C);
+
+ check_parallel_log($node_B, $offset_B, $streaming_mode, "prepared");
+ check_parallel_log($node_C, $offset_C, $streaming_mode, "prepared");
+
+ # check the transaction state is prepared on subscriber(s)
+ my $result =
+ $node_B->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(1), 'transaction is prepared on subscriber B');
+ $result =
+ $node_C->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(1), 'transaction is prepared on subscriber C');
+
+ # 2PC COMMIT
+ $node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';");
+
+ $node_A->wait_for_catchup($appname_B);
+ $node_B->wait_for_catchup($appname_C);
+
+ # check that transaction was committed on subscriber(s)
+ $result = $node_B->safe_psql('postgres',
+ "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+ is($result, qq(3334|3334|3334),
+ 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults'
+ );
+ $result = $node_C->safe_psql('postgres',
+ "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+ is($result, qq(3334|3334|3334),
+ 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults'
+ );
+
+ # check the transaction state is ended on subscriber(s)
+ $result =
+ $node_B->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(0), 'transaction is committed on subscriber B');
+ $result =
+ $node_C->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(0), 'transaction is committed on subscriber C');
+
+ ###############################
+ # Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT.
+ # 0. Cleanup from previous test leaving only 2 rows.
+ # 1. Insert one more row.
+ # 2. Record a SAVEPOINT.
+ # 3. Data is streamed using 2PC.
+ # 4. Do rollback to SAVEPOINT prior to the streamed inserts.
+ # 5. Then COMMIT PREPARED.
+ #
+ # Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1).
+ ###############################
+
+ # First, delete the data except for 2 rows (delete will be replicated)
+ $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;");
+
+ # Check the subscriber log from now on.
+ $offset_B = -s $node_B->logfile;
+ $offset_C = -s $node_C->logfile;
+
+ # 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT
+ $node_A->safe_psql(
+ 'postgres', "
+ BEGIN;
+ INSERT INTO test_tab VALUES (9999, 'foobar');
+ SAVEPOINT sp_inner;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+ UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+ DELETE FROM test_tab WHERE mod(a,3) = 0;
+ ROLLBACK TO SAVEPOINT sp_inner;
+ PREPARE TRANSACTION 'outer';
+ ");
+
+ $node_A->wait_for_catchup($appname_B);
+ $node_B->wait_for_catchup($appname_C);
+
+ check_parallel_log($node_B, $offset_B, $streaming_mode, "prepared");
+ check_parallel_log($node_C, $offset_C, $streaming_mode, "prepared");
+
+ # check the transaction state prepared on subscriber(s)
+ $result =
+ $node_B->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(1), 'transaction is prepared on subscriber B');
+ $result =
+ $node_C->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(1), 'transaction is prepared on subscriber C');
+
+ # 2PC COMMIT
+ $node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';");
+
+ $node_A->wait_for_catchup($appname_B);
+ $node_B->wait_for_catchup($appname_C);
+
+ # check the transaction state is ended on subscriber
+ $result =
+ $node_B->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(0), 'transaction is ended on subscriber B');
+ $result =
+ $node_C->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(0), 'transaction is ended on subscriber C');
+
+ # check inserts are visible at subscriber(s).
+ # All the streamed data (prior to the SAVEPOINT) should be rolled back.
+ # (9999, 'foobar') should be committed.
+ $result = $node_B->safe_psql('postgres',
+ "SELECT count(*) FROM test_tab where b = 'foobar';");
+ is($result, qq(1), 'Rows committed are present on subscriber B');
+ $result =
+ $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;");
+ is($result, qq(3), 'Rows committed are present on subscriber B');
+ $result = $node_C->safe_psql('postgres',
+ "SELECT count(*) FROM test_tab where b = 'foobar';");
+ is($result, qq(1), 'Rows committed are present on subscriber C');
+ $result =
+ $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;");
+ is($result, qq(3), 'Rows committed are present on subscriber C');
+
+ # Cleanup the test data
+ $node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;");
+ $node_A->wait_for_catchup($appname_B);
+ $node_B->wait_for_catchup($appname_C);
+
+ # Reset the log verbosity.
+ if ($streaming_mode eq 'parallel')
+ {
+ $node_B->append_conf('postgresql.conf', "log_min_messages = warning");
+ $node_B->reload;
+
+ $node_C->append_conf('postgresql.conf', "log_min_messages = warning");
+ $node_C->reload;
+ }
+}
+
###############################
# Setup a cascade of pub/sub nodes.
# node_A -> node_B -> node_C
@@ -260,160 +494,15 @@ is($result, qq(21), 'Rows committed are present on subscriber C');
# 2PC + STREAMING TESTS
# ---------------------
-my $oldpid_B = $node_A->safe_psql(
- 'postgres', "
- SELECT pid FROM pg_stat_replication
- WHERE application_name = '$appname_B' AND state = 'streaming';");
-my $oldpid_C = $node_B->safe_psql(
- 'postgres', "
- SELECT pid FROM pg_stat_replication
- WHERE application_name = '$appname_C' AND state = 'streaming';");
-
-# Setup logical replication (streaming = on)
-
-$node_B->safe_psql(
- 'postgres', "
- ALTER SUBSCRIPTION tap_sub_B
- SET (streaming = on);");
-$node_C->safe_psql(
- 'postgres', "
- ALTER SUBSCRIPTION tap_sub_C
- SET (streaming = on)");
-
-# Wait for subscribers to finish initialization
-
-$node_A->poll_query_until(
- 'postgres', "
- SELECT pid != $oldpid_B FROM pg_stat_replication
- WHERE application_name = '$appname_B' AND state = 'streaming';"
-) or die "Timed out while waiting for apply to restart";
-$node_B->poll_query_until(
- 'postgres', "
- SELECT pid != $oldpid_C FROM pg_stat_replication
- WHERE application_name = '$appname_C' AND state = 'streaming';"
-) or die "Timed out while waiting for apply to restart";
-
-###############################
-# Test 2PC PREPARE / COMMIT PREPARED.
-# 1. Data is streamed as a 2PC transaction.
-# 2. Then do commit prepared.
-#
-# Expect all data is replicated on subscriber(s) after the commit.
-###############################
-
-# Insert, update and delete enough rows to exceed the 64kB limit.
-# Then 2PC PREPARE
-$node_A->safe_psql(
- 'postgres', q{
- BEGIN;
- INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
- UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
- DELETE FROM test_tab WHERE mod(a,3) = 0;
- PREPARE TRANSACTION 'test_prepared_tab';});
-
-$node_A->wait_for_catchup($appname_B);
-$node_B->wait_for_catchup($appname_C);
+################################
+# Test using streaming mode 'on'
+################################
+test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on');
-# check the transaction state is prepared on subscriber(s)
-$result =
- $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(1), 'transaction is prepared on subscriber B');
-$result =
- $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(1), 'transaction is prepared on subscriber C');
-
-# 2PC COMMIT
-$node_A->safe_psql('postgres', "COMMIT PREPARED 'test_prepared_tab';");
-
-$node_A->wait_for_catchup($appname_B);
-$node_B->wait_for_catchup($appname_C);
-
-# check that transaction was committed on subscriber(s)
-$result = $node_B->safe_psql('postgres',
- "SELECT count(*), count(c), count(d = 999) FROM test_tab");
-is($result, qq(3334|3334|3334),
- 'Rows inserted by 2PC have committed on subscriber B, and extra columns have local defaults'
-);
-$result = $node_C->safe_psql('postgres',
- "SELECT count(*), count(c), count(d = 999) FROM test_tab");
-is($result, qq(3334|3334|3334),
- 'Rows inserted by 2PC have committed on subscriber C, and extra columns have local defaults'
-);
-
-# check the transaction state is ended on subscriber(s)
-$result =
- $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(0), 'transaction is committed on subscriber B');
-$result =
- $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(0), 'transaction is committed on subscriber C');
-
-###############################
-# Test 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT.
-# 0. Cleanup from previous test leaving only 2 rows.
-# 1. Insert one more row.
-# 2. Record a SAVEPOINT.
-# 3. Data is streamed using 2PC.
-# 4. Do rollback to SAVEPOINT prior to the streamed inserts.
-# 5. Then COMMIT PREPARED.
-#
-# Expect data after the SAVEPOINT is aborted leaving only 3 rows (= 2 original + 1 from step 1).
-###############################
-
-# First, delete the data except for 2 rows (delete will be replicated)
-$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;");
-
-# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT
-$node_A->safe_psql(
- 'postgres', "
- BEGIN;
- INSERT INTO test_tab VALUES (9999, 'foobar');
- SAVEPOINT sp_inner;
- INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
- UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
- DELETE FROM test_tab WHERE mod(a,3) = 0;
- ROLLBACK TO SAVEPOINT sp_inner;
- PREPARE TRANSACTION 'outer';
- ");
-
-$node_A->wait_for_catchup($appname_B);
-$node_B->wait_for_catchup($appname_C);
-
-# check the transaction state prepared on subscriber(s)
-$result =
- $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(1), 'transaction is prepared on subscriber B');
-$result =
- $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(1), 'transaction is prepared on subscriber C');
-
-# 2PC COMMIT
-$node_A->safe_psql('postgres', "COMMIT PREPARED 'outer';");
-
-$node_A->wait_for_catchup($appname_B);
-$node_B->wait_for_catchup($appname_C);
-
-# check the transaction state is ended on subscriber
-$result =
- $node_B->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(0), 'transaction is ended on subscriber B');
-$result =
- $node_C->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(0), 'transaction is ended on subscriber C');
-
-# check inserts are visible at subscriber(s).
-# All the streamed data (prior to the SAVEPOINT) should be rolled back.
-# (9999, 'foobar') should be committed.
-$result = $node_B->safe_psql('postgres',
- "SELECT count(*) FROM test_tab where b = 'foobar';");
-is($result, qq(1), 'Rows committed are present on subscriber B');
-$result = $node_B->safe_psql('postgres', "SELECT count(*) FROM test_tab;");
-is($result, qq(3), 'Rows committed are present on subscriber B');
-$result = $node_C->safe_psql('postgres',
- "SELECT count(*) FROM test_tab where b = 'foobar';");
-is($result, qq(1), 'Rows committed are present on subscriber C');
-$result = $node_C->safe_psql('postgres', "SELECT count(*) FROM test_tab;");
-is($result, qq(3), 'Rows committed are present on subscriber C');
+######################################
+# Test using streaming mode 'parallel'
+######################################
+test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'parallel');
###############################
# check all the cleanup
diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl
index 9b45410..a457957 100644
--- a/src/test/subscription/t/023_twophase_stream.pl
+++ b/src/test/subscription/t/023_twophase_stream.pl
@@ -8,6 +8,309 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
+# Check the log that the streamed transaction was completed successfully
+# reported by parallel apply worker.
+sub check_parallel_log
+{
+ my ($node_subscriber, $offset, $is_parallel, $action)= @_;
+ my $parallel_message = '';
+
+ if ($is_parallel)
+ {
+ $parallel_message = join("", 'streamed transaction \d+ was ', $action, ' using an parallel apply worker');
+
+ $node_subscriber->wait_for_log(qr/$parallel_message/, $offset);
+ }
+}
+
+# Encapsulate all the common test steps which are related to "streaming"
+# parameter so the same code can be run both for the streaming=on and
+# streaming=parallel cases.
+sub test_streaming
+{
+ my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
+
+ my $offset = 0;
+
+ ###############################
+ # Test 2PC PREPARE / COMMIT PREPARED.
+ # 1. Data is streamed as a 2PC transaction.
+ # 2. Then do commit prepared.
+ #
+ # Expect all data is replicated on subscriber side after the commit.
+ ###############################
+
+ # If "streaming" parameter is specified as "parallel", we need to check
+ # that streamed transaction was prepared using an parallel apply worker.
+ # We have to look for the DEBUG1 log messages about that, so bump up the
+ # log verbosity.
+ if ($is_parallel)
+ {
+ $node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = debug1");
+ $node_subscriber->reload;
+ }
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ # check that 2PC gets replicated to subscriber
+ # Insert, update and delete enough rows to exceed the 64kB limit.
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+ UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+ DELETE FROM test_tab WHERE mod(a,3) = 0;
+ PREPARE TRANSACTION 'test_prepared_tab';});
+
+ $node_publisher->wait_for_catchup($appname);
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "prepared");
+
+ # check that transaction is in prepared state on subscriber
+ my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(1), 'transaction is prepared on subscriber');
+
+ # 2PC transaction gets committed
+ $node_publisher->safe_psql('postgres',
+ "COMMIT PREPARED 'test_prepared_tab';");
+
+ $node_publisher->wait_for_catchup($appname);
+
+ # check that transaction is committed on subscriber
+ $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+ is($result, qq(3334|3334|3334),
+ 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults'
+ );
+ $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(0), 'transaction is committed on subscriber');
+
+ ###############################
+ # Test 2PC PREPARE / ROLLBACK PREPARED.
+ # 1. Table is deleted back to 2 rows which are replicated on subscriber.
+ # 2. Data is streamed using 2PC.
+ # 3. Do rollback prepared.
+ #
+ # Expect data rolls back leaving only the original 2 rows.
+ ###############################
+
+ # First, delete the data except for 2 rows (will be replicated)
+ $node_publisher->safe_psql('postgres',
+ "DELETE FROM test_tab WHERE a > 2;");
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ # Then insert, update and delete enough rows to exceed the 64kB limit.
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+ UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+ DELETE FROM test_tab WHERE mod(a,3) = 0;
+ PREPARE TRANSACTION 'test_prepared_tab';});
+
+ $node_publisher->wait_for_catchup($appname);
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "prepared");
+
+ # check that transaction is in prepared state on subscriber
+ $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(1), 'transaction is prepared on subscriber');
+
+ # 2PC transaction gets aborted
+ $node_publisher->safe_psql('postgres',
+ "ROLLBACK PREPARED 'test_prepared_tab';");
+
+ $node_publisher->wait_for_catchup($appname);
+
+ # check that transaction is aborted on subscriber
+ $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+ is($result, qq(2|2|2),
+ 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows'
+ );
+
+ $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(0), 'transaction is aborted on subscriber');
+
+ ###############################
+ # Check that 2PC COMMIT PREPARED is decoded properly on crash restart.
+ # 1. insert, update and delete enough rows to exceed the 64kB limit.
+ # 2. Then server crashes before the 2PC transaction is committed.
+ # 3. After servers are restarted the pending transaction is committed.
+ #
+ # Expect all data is replicated on subscriber side after the commit.
+ # Note: both publisher and subscriber do crash/restart.
+ ###############################
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+ UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+ DELETE FROM test_tab WHERE mod(a,3) = 0;
+ PREPARE TRANSACTION 'test_prepared_tab';});
+
+ $node_subscriber->stop('immediate');
+ $node_publisher->stop('immediate');
+
+ $node_publisher->start;
+ $node_subscriber->start;
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "prepared");
+
+ # commit post the restart
+ $node_publisher->safe_psql('postgres',
+ "COMMIT PREPARED 'test_prepared_tab';");
+ $node_publisher->wait_for_catchup($appname);
+
+ # check inserts are visible
+ $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+ is($result, qq(3334|3334|3334),
+ 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults'
+ );
+
+ ###############################
+ # Do INSERT after the PREPARE but before ROLLBACK PREPARED.
+ # 1. Table is deleted back to 2 rows which are replicated on subscriber.
+ # 2. Data is streamed using 2PC.
+ # 3. A single row INSERT is done which is after the PREPARE.
+ # 4. Then do a ROLLBACK PREPARED.
+ #
+ # Expect the 2PC data rolls back leaving only 3 rows on the subscriber
+ # (the original 2 + inserted 1).
+ ###############################
+
+ # First, delete the data except for 2 rows (will be replicated)
+ $node_publisher->safe_psql('postgres',
+ "DELETE FROM test_tab WHERE a > 2;");
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ # Then insert, update and delete enough rows to exceed the 64kB limit.
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+ UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+ DELETE FROM test_tab WHERE mod(a,3) = 0;
+ PREPARE TRANSACTION 'test_prepared_tab';});
+
+ $node_publisher->wait_for_catchup($appname);
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "prepared");
+
+ # check that transaction is in prepared state on subscriber
+ $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(1), 'transaction is prepared on subscriber');
+
+ # Insert a different record (now we are outside of the 2PC transaction)
+ # Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key
+ $node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (99999, 'foobar')");
+
+ # 2PC transaction gets aborted
+ $node_publisher->safe_psql('postgres',
+ "ROLLBACK PREPARED 'test_prepared_tab';");
+
+ $node_publisher->wait_for_catchup($appname);
+
+ # check that transaction is aborted on subscriber,
+ # but the extra INSERT outside of the 2PC still was replicated
+ $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+ is($result, qq(3|3|3),
+ 'check the outside insert was copied to subscriber');
+
+ $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(0), 'transaction is aborted on subscriber');
+
+ ###############################
+ # Do INSERT after the PREPARE but before COMMIT PREPARED.
+ # 1. Table is deleted back to 2 rows which are replicated on subscriber.
+ # 2. Data is streamed using 2PC.
+ # 3. A single row INSERT is done which is after the PREPARE.
+ # 4. Then do a COMMIT PREPARED.
+ #
+ # Expect 2PC data + the extra row are on the subscriber
+ # (the 3334 + inserted 1 = 3335).
+ ###############################
+
+ # First, delete the data except for 2 rows (will be replicated)
+ $node_publisher->safe_psql('postgres',
+ "DELETE FROM test_tab WHERE a > 2;");
+
+ # Check the subscriber log from now on.
+ $offset = -s $node_subscriber->logfile;
+
+ # Then insert, update and delete enough rows to exceed the 64kB limit.
+ $node_publisher->safe_psql(
+ 'postgres', q{
+ BEGIN;
+ INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+ UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+ DELETE FROM test_tab WHERE mod(a,3) = 0;
+ PREPARE TRANSACTION 'test_prepared_tab';});
+
+ $node_publisher->wait_for_catchup($appname);
+
+ check_parallel_log($node_subscriber, $offset, $is_parallel, "prepared");
+
+ # check that transaction is in prepared state on subscriber
+ $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(1), 'transaction is prepared on subscriber');
+
+ # Insert a different record (now we are outside of the 2PC transaction)
+ # Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key
+ $node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (99999, 'foobar')");
+
+ # 2PC transaction gets committed
+ $node_publisher->safe_psql('postgres',
+ "COMMIT PREPARED 'test_prepared_tab';");
+
+ $node_publisher->wait_for_catchup($appname);
+
+ # check that transaction is committed on subscriber
+ $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+ is($result, qq(3335|3335|3335),
+ 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults'
+ );
+
+ $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_prepared_xacts;");
+ is($result, qq(0), 'transaction is committed on subscriber');
+
+ # Cleanup the test data
+ $node_publisher->safe_psql('postgres',
+ "DELETE FROM test_tab WHERE a > 2;");
+ $node_publisher->wait_for_catchup($appname);
+
+ # Reset the log verbosity.
+ if ($is_parallel)
+ {
+ $node_subscriber->append_conf('postgresql.conf',
+ "log_min_messages = warning");
+ $node_subscriber->reload;
+ }
+}
+
###############################
# Setup
###############################
@@ -48,6 +351,10 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR TABLE test_tab");
my $appname = 'tap_sub';
+
+################################
+# Test using streaming mode 'on'
+################################
$node_subscriber->safe_psql(
'postgres', "
CREATE SUBSCRIPTION tap_sub
@@ -64,236 +371,30 @@ my $twophase_query =
$node_subscriber->poll_query_until('postgres', $twophase_query)
or die "Timed out while waiting for subscriber to enable twophase";
-###############################
# Check initial data was copied to subscriber
-###############################
my $result = $node_subscriber->safe_psql('postgres',
"SELECT count(*), count(c), count(d = 999) FROM test_tab");
is($result, qq(2|2|2), 'check initial data was copied to subscriber');
-###############################
-# Test 2PC PREPARE / COMMIT PREPARED.
-# 1. Data is streamed as a 2PC transaction.
-# 2. Then do commit prepared.
-#
-# Expect all data is replicated on subscriber side after the commit.
-###############################
+test_streaming($node_publisher, $node_subscriber, $appname, 0);
-# check that 2PC gets replicated to subscriber
-# Insert, update and delete enough rows to exceed the 64kB limit.
-$node_publisher->safe_psql(
- 'postgres', q{
- BEGIN;
- INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
- UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
- DELETE FROM test_tab WHERE mod(a,3) = 0;
- PREPARE TRANSACTION 'test_prepared_tab';});
-
-$node_publisher->wait_for_catchup($appname);
-
-# check that transaction is in prepared state on subscriber
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(1), 'transaction is prepared on subscriber');
-
-# 2PC transaction gets committed
-$node_publisher->safe_psql('postgres',
- "COMMIT PREPARED 'test_prepared_tab';");
-
-$node_publisher->wait_for_catchup($appname);
-
-# check that transaction is committed on subscriber
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(c), count(d = 999) FROM test_tab");
-is($result, qq(3334|3334|3334),
- 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults'
+######################################
+# Test using streaming mode 'parallel'
+######################################
+my $oldpid = $node_publisher->safe_psql('postgres',
+ "SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
);
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(0), 'transaction is committed on subscriber');
-
-###############################
-# Test 2PC PREPARE / ROLLBACK PREPARED.
-# 1. Table is deleted back to 2 rows which are replicated on subscriber.
-# 2. Data is streamed using 2PC.
-# 3. Do rollback prepared.
-#
-# Expect data rolls back leaving only the original 2 rows.
-###############################
-
-# First, delete the data except for 2 rows (will be replicated)
-$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;");
-
-# Then insert, update and delete enough rows to exceed the 64kB limit.
-$node_publisher->safe_psql(
- 'postgres', q{
- BEGIN;
- INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
- UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
- DELETE FROM test_tab WHERE mod(a,3) = 0;
- PREPARE TRANSACTION 'test_prepared_tab';});
-$node_publisher->wait_for_catchup($appname);
-
-# check that transaction is in prepared state on subscriber
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(1), 'transaction is prepared on subscriber');
-
-# 2PC transaction gets aborted
-$node_publisher->safe_psql('postgres',
- "ROLLBACK PREPARED 'test_prepared_tab';");
-
-$node_publisher->wait_for_catchup($appname);
-
-# check that transaction is aborted on subscriber
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(c), count(d = 999) FROM test_tab");
-is($result, qq(2|2|2),
- 'Rows inserted by 2PC are rolled back, leaving only the original 2 rows');
-
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(0), 'transaction is aborted on subscriber');
-
-###############################
-# Check that 2PC COMMIT PREPARED is decoded properly on crash restart.
-# 1. insert, update and delete enough rows to exceed the 64kB limit.
-# 2. Then server crashes before the 2PC transaction is committed.
-# 3. After servers are restarted the pending transaction is committed.
-#
-# Expect all data is replicated on subscriber side after the commit.
-# Note: both publisher and subscriber do crash/restart.
-###############################
-
-$node_publisher->safe_psql(
- 'postgres', q{
- BEGIN;
- INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
- UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
- DELETE FROM test_tab WHERE mod(a,3) = 0;
- PREPARE TRANSACTION 'test_prepared_tab';});
-
-$node_subscriber->stop('immediate');
-$node_publisher->stop('immediate');
-
-$node_publisher->start;
-$node_subscriber->start;
-
-# commit post the restart
-$node_publisher->safe_psql('postgres',
- "COMMIT PREPARED 'test_prepared_tab';");
-$node_publisher->wait_for_catchup($appname);
-
-# check inserts are visible
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(c), count(d = 999) FROM test_tab");
-is($result, qq(3334|3334|3334),
- 'Rows inserted by 2PC have committed on subscriber, and extra columns contain local defaults'
-);
-
-###############################
-# Do INSERT after the PREPARE but before ROLLBACK PREPARED.
-# 1. Table is deleted back to 2 rows which are replicated on subscriber.
-# 2. Data is streamed using 2PC.
-# 3. A single row INSERT is done which is after the PREPARE.
-# 4. Then do a ROLLBACK PREPARED.
-#
-# Expect the 2PC data rolls back leaving only 3 rows on the subscriber
-# (the original 2 + inserted 1).
-###############################
-
-# First, delete the data except for 2 rows (will be replicated)
-$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;");
-
-# Then insert, update and delete enough rows to exceed the 64kB limit.
-$node_publisher->safe_psql(
- 'postgres', q{
- BEGIN;
- INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
- UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
- DELETE FROM test_tab WHERE mod(a,3) = 0;
- PREPARE TRANSACTION 'test_prepared_tab';});
-
-$node_publisher->wait_for_catchup($appname);
-
-# check that transaction is in prepared state on subscriber
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(1), 'transaction is prepared on subscriber');
-
-# Insert a different record (now we are outside of the 2PC transaction)
-# Note: the 2PC transaction still holds row locks so make sure this insert is for a separate primary key
-$node_publisher->safe_psql('postgres',
- "INSERT INTO test_tab VALUES (99999, 'foobar')");
-
-# 2PC transaction gets aborted
-$node_publisher->safe_psql('postgres',
- "ROLLBACK PREPARED 'test_prepared_tab';");
-
-$node_publisher->wait_for_catchup($appname);
-
-# check that transaction is aborted on subscriber,
-# but the extra INSERT outside of the 2PC still was replicated
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(c), count(d = 999) FROM test_tab");
-is($result, qq(3|3|3), 'check the outside insert was copied to subscriber');
-
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(0), 'transaction is aborted on subscriber');
-
-###############################
-# Do INSERT after the PREPARE but before COMMIT PREPARED.
-# 1. Table is deleted back to 2 rows which are replicated on subscriber.
-# 2. Data is streamed using 2PC.
-# 3. A single row INSERT is done which is after the PREPARE.
-# 4. Then do a COMMIT PREPARED.
-#
-# Expect 2PC data + the extra row are on the subscriber
-# (the 3334 + inserted 1 = 3335).
-###############################
-
-# First, delete the data except for 2 rows (will be replicated)
-$node_publisher->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;");
-
-# Then insert, update and delete enough rows to exceed the 64kB limit.
-$node_publisher->safe_psql(
- 'postgres', q{
- BEGIN;
- INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
- UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
- DELETE FROM test_tab WHERE mod(a,3) = 0;
- PREPARE TRANSACTION 'test_prepared_tab';});
-
-$node_publisher->wait_for_catchup($appname);
-
-# check that transaction is in prepared state on subscriber
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(1), 'transaction is prepared on subscriber');
-
-# Insert a different record (now we are outside of the 2PC transaction)
-# Note: the 2PC transaction still holds row locks so make sure this insert is for a separare primary key
-$node_publisher->safe_psql('postgres',
- "INSERT INTO test_tab VALUES (99999, 'foobar')");
-
-# 2PC transaction gets committed
-$node_publisher->safe_psql('postgres',
- "COMMIT PREPARED 'test_prepared_tab';");
-
-$node_publisher->wait_for_catchup($appname);
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)");
-# check that transaction is committed on subscriber
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT count(*), count(c), count(d = 999) FROM test_tab");
-is($result, qq(3335|3335|3335),
- 'Rows inserted by 2PC (as well as outside insert) have committed on subscriber, and extra columns contain local defaults'
-);
+$node_publisher->poll_query_until('postgres',
+ "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
+ )
+ or die
+ "Timed out while waiting for apply to restart after changing SUBSCRIPTION";
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(0), 'transaction is committed on subscriber');
+test_streaming($node_publisher, $node_subscriber, $appname, 1);
###############################
# check all the cleanup
--
2.7.2.windows.1
[application/octet-stream] v26-0003-Add-some-checks-before-using-parallel-apply-work.patch (51.3K, ../OS0PR01MB5716546863340CA19EC5F8C394789@OS0PR01MB5716.jpnprd01.prod.outlook.com/5-v26-0003-Add-some-checks-before-using-parallel-apply-work.patch)
download | inline diff:
From 213590a664e34a34965465662d55e6737af90d37 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <[email protected]>
Date: Wed, 10 Aug 2022 20:37:22 +0800
Subject: [PATCH v26 3/5] Add some checks before using parallel apply worker to
apply changes
streaming=parallel mode has three requirements:
1) The unique column in the relation on the subscriber-side should also be the
unique column on the publisher-side;
2) There cannot be any non-immutable functions used by the subscriber-side
replicated table. Look for functions in the following places:
* a. Trigger functions
* b. Column default value expressions and domain constraints
* c. Constraint expressions
3) The foreign keys are not allowed.
It is better to only check the foreign key when it is detected that the
publisher does not have the foreign key and the subscriber does. But this
requires the publisher to send more information. In addition, since foreign key
does not take effect in the subscriber's apply worker by default, it seems that
foreign key does not hit this ERROR frequently. So, only perform a simple check
based on the subscriber schema.
Without these safety checks, the following scenario may occur:
The parallel apply worker locks a row when processing a streaming transaction,
after that the leader apply worker tries to lock the same row when processing
another non-streamed transaction. At this time, the leader apply worker waits
for the streaming transaction to complete and the lock to be released, it won't
send subsequent data of the streaming transaction to the parallel apply worker;
the parallel apply worker waits to receive the rest of streaming transaction
and can't finish this transaction. Now a deadlock has occurred, so both workers
will wait indefinitely.
---
doc/src/sgml/ref/create_subscription.sgml | 6 +
.../replication/logical/applyparallelworker.c | 75 +++
src/backend/replication/logical/proto.c | 86 ++-
src/backend/replication/logical/relation.c | 215 ++++++++
src/backend/replication/logical/tablesync.c | 1 +
src/backend/replication/logical/worker.c | 23 +-
src/backend/utils/cache/typcache.c | 17 +
src/include/replication/logicalproto.h | 1 +
src/include/replication/logicalrelation.h | 15 +
src/include/replication/worker_internal.h | 3 +
src/include/utils/typcache.h | 2 +
src/test/subscription/t/015_stream.pl | 9 +-
src/test/subscription/t/016_stream_subxact.pl | 9 +-
src/test/subscription/t/022_twophase_cascade.pl | 8 +
src/test/subscription/t/023_twophase_stream.pl | 9 +-
.../t/032_streaming_parallel_safety.pl | 608 +++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
17 files changed, 1073 insertions(+), 15 deletions(-)
create mode 100644 src/test/subscription/t/032_streaming_parallel_safety.pl
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index cbcd02b..325eee8 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -240,6 +240,12 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
transaction is committed. Note that if an error happens when
applying changes in a parallel worker, the finish LSN of the
remote transaction might not be reported in the server log.
+ There are three prerequisites for using <literal>parallel</literal>
+ mode: 1) the unique column in the table on the subscriber-side must
+ also be the unique column on the publisher-side; 2) there cannot be
+ any non-immutable functions used by the subscriber-side replicated
+ table. 3) there cannot be any foreign keys used by the
+ subscriber-side replicated table.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index f592c2b..dea19bb 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -1087,3 +1087,78 @@ parallel_apply_replorigin_reset(void)
if (started_tx)
CommitTransactionCommand();
}
+
+/*
+ * Check if any parallel apply workers are in use.
+ */
+bool
+parallel_apply_using_worker(void)
+{
+ ListCell *lc;
+
+ /* Return true for parallel apply worker */
+ if (am_parallel_apply_worker())
+ return true;
+
+ foreach(lc, ParallelApplyWorkersList)
+ {
+ ParallelApplyWorkerInfo *tmp_winfo;
+
+ tmp_winfo = (ParallelApplyWorkerInfo *) lfirst(lc);
+
+ if (tmp_winfo->in_remote_transaction)
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Check if changes on this relation can be applied using an parallel apply
+ * worker.
+ *
+ * Although the commit order is maintained by only allowing one process to
+ * commit at a time, the access order to the relation has changed. This could
+ * cause unexpected problems when applying transactions using an parallel
+ * apply worker if the unique column on the replicated table is
+ * inconsistent with the publisher-side, or if the relation contains
+ * non-immutable functions.
+ */
+void
+parallel_apply_relation_check(LogicalRepRelMapEntry *rel)
+{
+ /*
+ * Skip check if not using parallel apply workers.
+ *
+ * If any worker is handling the streaming transaction, this check not only
+ * needs to be performed by parallel apply worker, but also by the leader
+ * apply worker. This is because without these safety checks, leader apply
+ * worker may block parallel apply worker, which will cause infinite
+ * waits.
+ */
+ if (!parallel_apply_using_worker())
+ return;
+
+ /*
+ * Partition table checks are done later in function
+ * apply_handle_tuple_routing.
+ */
+ if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ return;
+
+ if (rel->parallel_apply_safe == PARALLEL_APPLY_SAFETY_UNKNOWN)
+ logicalrep_rel_mark_parallel_apply(rel);
+
+ if (rel->parallel_apply_safe == PARALLEL_APPLY_UNSAFE)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot replicate target relation \"%s.%s\" using "
+ "subscription parameter %s",
+ rel->remoterel.nspname, rel->remoterel.relname,
+ "streaming = parallel"),
+ errdetail("The unique column on subscriber is not the unique "
+ "column on publisher or there is at least one "
+ "non-immutable function."),
+ errhint("Please change to use subscription parameter %s.",
+ "streaming = on")));
+}
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 47bd811..73b2807 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -23,7 +23,8 @@
/*
* Protocol message flags.
*/
-#define LOGICALREP_IS_REPLICA_IDENTITY 1
+#define ATTR_IS_REPLICA_IDENTITY (1 << 0)
+#define ATTR_IS_UNIQUE (1 << 1)
#define MESSAGE_TRANSACTIONAL (1<<0)
#define TRUNCATE_CASCADE (1<<0)
@@ -41,6 +42,66 @@ static void logicalrep_write_namespace(StringInfo out, Oid nspid);
static const char *logicalrep_read_namespace(StringInfo in);
/*
+ * RelationGetUniqueKeyBitmap -- get a bitmap of unique attribute numbers
+ *
+ * This is similar to RelationGetIdentityKeyBitmap(), but returns a bitmap of
+ * index attribute numbers for all unique indexes.
+ */
+static Bitmapset *
+RelationGetUniqueKeyBitmap(Relation rel)
+{
+ List *indexoidlist = NIL;
+ ListCell *indexoidscan;
+ Bitmapset *attunique = NULL;
+
+ if (!rel->rd_rel->relhasindex)
+ return NULL;
+
+ indexoidlist = RelationGetIndexList(rel);
+
+ foreach(indexoidscan, indexoidlist)
+ {
+ Oid indexoid = lfirst_oid(indexoidscan);
+ Relation indexRel;
+ int i;
+
+ /* Look up the description for index */
+ indexRel = RelationIdGetRelation(indexoid);
+
+ if (!RelationIsValid(indexRel))
+ elog(ERROR, "could not open relation with OID %u", indexoid);
+
+ if (!indexRel->rd_index->indisunique)
+ {
+ RelationClose(indexRel);
+ continue;
+ }
+
+ /* Add referenced attributes to attunique */
+ for (i = 0; i < indexRel->rd_index->indnatts; i++)
+ {
+ int attrnum = indexRel->rd_index->indkey.values[i];
+
+ /*
+ * We don't include non-key columns into attunique
+ * bitmaps. See RelationGetIndexAttrBitmap.
+ */
+ if (attrnum != 0)
+ {
+ if (i < indexRel->rd_index->indnkeyatts &&
+ !bms_is_member(attrnum - FirstLowInvalidHeapAttributeNumber, attunique))
+ attunique = bms_add_member(attunique,
+ attrnum - FirstLowInvalidHeapAttributeNumber);
+ }
+ }
+ RelationClose(indexRel);
+ }
+ list_free(indexoidlist);
+
+ return attunique;
+}
+
+/*
* Check if a column is covered by a column list.
*
* Need to be careful about NULL, which is treated as a column list covering
@@ -933,7 +994,8 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
TupleDesc desc;
int i;
uint16 nliveatts = 0;
- Bitmapset *idattrs = NULL;
+ Bitmapset *idattrs = NULL,
+ *attunique = NULL;
bool replidentfull;
desc = RelationGetDescr(rel);
@@ -958,6 +1020,9 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
if (!replidentfull)
idattrs = RelationGetIdentityKeyBitmap(rel);
+ /* fetch bitmap of UNIQUE attributes */
+ attunique = RelationGetUniqueKeyBitmap(rel);
+
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -974,7 +1039,11 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
idattrs))
- flags |= LOGICALREP_IS_REPLICA_IDENTITY;
+ flags |= ATTR_IS_REPLICA_IDENTITY;
+
+ if (bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ attunique))
+ flags |= ATTR_IS_UNIQUE;
pq_sendbyte(out, flags);
@@ -989,6 +1058,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
}
bms_free(idattrs);
+ bms_free(attunique);
}
/*
@@ -1001,7 +1071,8 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel)
int natts;
char **attnames;
Oid *atttyps;
- Bitmapset *attkeys = NULL;
+ Bitmapset *attkeys = NULL,
+ *attunique = NULL;
natts = pq_getmsgint(in, 2);
attnames = palloc(natts * sizeof(char *));
@@ -1014,9 +1085,13 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel)
/* Check for replica identity column */
flags = pq_getmsgbyte(in);
- if (flags & LOGICALREP_IS_REPLICA_IDENTITY)
+ if (flags & ATTR_IS_REPLICA_IDENTITY)
attkeys = bms_add_member(attkeys, i);
+ /* Check for unique column */
+ if (flags & ATTR_IS_UNIQUE)
+ attunique = bms_add_member(attunique, i);
+
/* attribute name */
attnames[i] = pstrdup(pq_getmsgstring(in));
@@ -1030,6 +1105,7 @@ logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel)
rel->attnames = attnames;
rel->atttyps = atttyps;
rel->attkeys = attkeys;
+ rel->attunique = attunique;
rel->natts = natts;
}
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e989047..fc175a8 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -19,12 +19,19 @@
#include "access/table.h"
#include "catalog/namespace.h"
+#include "catalog/pg_proc.h"
#include "catalog/pg_subscription_rel.h"
+#include "commands/trigger.h"
#include "executor/executor.h"
#include "nodes/makefuncs.h"
+#include "optimizer/optimizer.h"
#include "replication/logicalrelation.h"
#include "replication/worker_internal.h"
+#include "rewrite/rewriteHandler.h"
#include "utils/inval.h"
+#include "utils/lsyscache.h"
+#include "utils/syscache.h"
+#include "utils/typcache.h"
static MemoryContext LogicalRepRelMapContext = NULL;
@@ -92,6 +99,23 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
/*
+ * Syscache invalidation callback to reset parallel_apply_safe flag.
+ */
+static void
+logicalrep_relmap_reset_parallel_cb(Datum arg, int cacheid, uint32 hashvalue)
+{
+ HASH_SEQ_STATUS hash_seq;
+ LogicalRepRelMapEntry *entry;
+
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ hash_seq_init(&hash_seq, LogicalRepRelMap);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ entry->parallel_apply_safe = PARALLEL_APPLY_SAFETY_UNKNOWN;
+}
+
+/*
* Initialize the relation map cache.
*/
static void
@@ -116,6 +140,12 @@ logicalrep_relmap_init(void)
/* Watch for invalidation events. */
CacheRegisterRelcacheCallback(logicalrep_relmap_invalidate_cb,
(Datum) 0);
+ CacheRegisterSyscacheCallback(PROCOID,
+ logicalrep_relmap_reset_parallel_cb,
+ (Datum) 0);
+ CacheRegisterSyscacheCallback(TYPEOID,
+ logicalrep_relmap_reset_parallel_cb,
+ (Datum) 0);
}
/*
@@ -142,6 +172,7 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry)
pfree(remoterel->atttyps);
}
bms_free(remoterel->attkeys);
+ bms_free(remoterel->attunique);
if (entry->attrmap)
free_attrmap(entry->attrmap);
@@ -190,6 +221,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel)
}
entry->remoterel.replident = remoterel->replident;
entry->remoterel.attkeys = bms_copy(remoterel->attkeys);
+ entry->remoterel.attunique = bms_copy(remoterel->attunique);
MemoryContextSwitchTo(oldctx);
}
@@ -311,6 +343,182 @@ logicalrep_rel_mark_updatable(LogicalRepRelMapEntry *entry)
}
/*
+ * Check if changes on one relation can be applied using an parallel apply
+ * worker and assign the 'parallel_apply_safe' flag.
+ *
+ * There are three requirements for applying changes using an parallel apply
+ * worker: 1) the unique column in the table on the subscriber-side should also
+ * be the unique column on the publisher-side; 2) there cannot be any
+ * non-immutable functions used by the subscriber-side replicated table. 3)
+ * there cannot be any foreign keys used by the subscriber-side replicated
+ * table.
+ *
+ * Without these safety checks, the following scenario may occur: The parallel
+ * apply worker locks a row when processing a streaming transaction, after
+ * that the leader apply worker tries to lock the same row when processing
+ * another non-streamed transaction. At this time, the leader apply worker
+ * waits for the streaming transaction to complete and the lock to be released,
+ * it won't send subsequent data of the streaming transaction to the parallel
+ * apply worker; the parallel apply worker waits to receive the rest of
+ * streaming transaction and can't finish this transaction. Now a deadlock has
+ * occurred, so both workers will wait indefinitely.
+ *
+ * We just mark the relation entry as 'PARALLEL_APPLY_UNSAFE' here if changes
+ * on one relation can not be applied using an parallel apply worker and leave
+ * it to parallel_apply_relation_check() to throw the actual error if needed.
+ */
+void
+logicalrep_rel_mark_parallel_apply(LogicalRepRelMapEntry *entry)
+{
+ Bitmapset *ukey;
+ int i;
+ TupleDesc tupdesc;
+ int attnum;
+ List *fkeys = NIL;
+
+ /* Skip if no parallel apply worker is in use */
+ if (!parallel_apply_using_worker())
+ return;
+
+ /* Initialize the flag. */
+ entry->parallel_apply_safe = PARALLEL_APPLY_SAFETY_UNKNOWN;
+
+ /*
+ * First, check if the unique column in the relation on the subscriber-side
+ * is also the unique column on the publisher-side.
+ */
+ ukey = RelationGetIndexAttrBitmap(entry->localrel,
+ INDEX_ATTR_BITMAP_KEY);
+
+ if (ukey)
+ {
+ i = -1;
+ while ((i = bms_next_member(ukey, i)) >= 0)
+ {
+ attnum = AttrNumberGetAttrOffset(i + FirstLowInvalidHeapAttributeNumber);
+
+ if (entry->attrmap->attnums[attnum] < 0 ||
+ !bms_is_member(entry->attrmap->attnums[attnum], entry->remoterel.attunique))
+ {
+ entry->parallel_apply_safe = PARALLEL_APPLY_UNSAFE;
+ bms_free(ukey);
+ return;
+ }
+ }
+
+ bms_free(ukey);
+ }
+
+ /*
+ * Then, check if there is any non-immutable function used by the
+ * subscriber-side relation. Look for functions in the following places:
+ * a. trigger functions;
+ * b. Column default value expressions and domain constraints;
+ * c. Constraint expressions;
+ */
+ /* Check the trigger functions. */
+ if (entry->localrel->trigdesc != NULL)
+ {
+ for (i = 0; i < entry->localrel->trigdesc->numtriggers; i++)
+ {
+ Trigger *trig = entry->localrel->trigdesc->triggers + i;
+
+ if (trig->tgenabled != TRIGGER_FIRES_ALWAYS &&
+ trig->tgenabled != TRIGGER_FIRES_ON_REPLICA)
+ continue;
+
+ if (func_volatile(trig->tgfoid) != PROVOLATILE_IMMUTABLE)
+ {
+ entry->parallel_apply_safe = PARALLEL_APPLY_UNSAFE;
+ return;
+ }
+ }
+ }
+
+ /* Check the columns. */
+ tupdesc = RelationGetDescr(entry->localrel);
+ for (attnum = 0; attnum < tupdesc->natts; attnum++)
+ {
+ Form_pg_attribute att = TupleDescAttr(tupdesc, attnum);
+ Node *defaultexpr = NULL;
+
+ /* We don't check dropped or generated attributes */
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ defaultexpr = build_column_default(entry->localrel, attnum + 1);
+ if (defaultexpr && contain_mutable_functions(defaultexpr))
+ {
+ entry->parallel_apply_safe = PARALLEL_APPLY_UNSAFE;
+ return;
+ }
+
+ /*
+ * If the column is of a DOMAIN type, determine whether that domain has
+ * any CHECK expressions that are not immutable.
+ */
+ if (get_typtype(att->atttypid) == TYPTYPE_DOMAIN)
+ {
+ List *domain_constraints;
+ ListCell *lc;
+
+ domain_constraints = GetDomainConstraints(att->atttypid);
+
+ foreach(lc, domain_constraints)
+ {
+ DomainConstraintState *con = (DomainConstraintState *) lfirst(lc);
+
+ if (con->check_expr && contain_mutable_functions((Node *) con->check_expr))
+ {
+ entry->parallel_apply_safe = PARALLEL_APPLY_UNSAFE;
+ return;
+ }
+ }
+ }
+ }
+
+ /* Check the constraints. */
+ if (tupdesc->constr)
+ {
+ ConstrCheck *check = tupdesc->constr->check;
+
+ /*
+ * Determine if there are any CHECK constraints which
+ * contains non-immutable function.
+ */
+ for (i = 0; i < tupdesc->constr->num_check; i++)
+ {
+ Expr *check_expr = stringToNode(check[i].ccbin);
+
+ if (contain_mutable_functions((Node *) check_expr))
+ {
+ entry->parallel_apply_safe = PARALLEL_APPLY_UNSAFE;
+ return;
+ }
+ }
+ }
+
+ /*
+ * Finally, check the foreign keys.
+ *
+ * XXX It is better to only check the foreign key when it is detected that
+ * the publisher does not have the foreign key and the subscriber does. But
+ * this requires the publisher to send more information. In addition, since
+ * foreign key does not take effect in the subscriber's apply worker by
+ * default, it seems that foreign key does not hit this ERROR frequently.
+ * So, only a simple check based on the subscriber schema is kept here.
+ */
+ fkeys = RelationGetFKeyList(entry->localrel);
+ if (fkeys)
+ {
+ entry->parallel_apply_safe = PARALLEL_APPLY_UNSAFE;
+ return;
+ }
+
+ entry->parallel_apply_safe = PARALLEL_APPLY_SAFE;
+}
+
+/*
* Open the local relation associated with the remote one.
*
* Rebuilds the Relcache mapping if it was invalidated by local DDL.
@@ -438,6 +646,9 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
*/
logicalrep_rel_mark_updatable(entry);
+ /* Set if changes could be applied using an parallel apply worker */
+ logicalrep_rel_mark_parallel_apply(entry);
+
entry->localrelvalid = true;
}
@@ -653,6 +864,7 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root,
}
entry->remoterel.replident = remoterel->replident;
entry->remoterel.attkeys = bms_copy(remoterel->attkeys);
+ entry->remoterel.attunique = bms_copy(remoterel->attunique);
}
entry->localrel = partrel;
@@ -696,6 +908,9 @@ logicalrep_partition_open(LogicalRepRelMapEntry *root,
/* Set if the table's replica identity is enough to apply update/delete. */
logicalrep_rel_mark_updatable(entry);
+ /* Set if changes could be applied using an parallel apply worker */
+ logicalrep_rel_mark_parallel_apply(entry);
+
entry->localrelvalid = true;
/* state and statelsn are left set to 0. */
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 60dfb86..170997a 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -880,6 +880,7 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *));
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ lrel->attunique = NULL;
/*
* Store the columns as a list of names. Ignore those that are not
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c191797..a7e1ea2 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1471,6 +1471,14 @@ apply_handle_stream_stop(StringInfo s)
{
Assert(winfo);
+ /*
+ * Unlike stream_commit, we don't need to wait here for stream_stop to
+ * finish. Allowing the other transaction to be applied before
+ * stream_stop is finished can lead to failures if the unique
+ * index/constraint is different between publisher and subscriber. But
+ * for such cases, we don't allow streamed transactions to be applied
+ * in parallel. See parallel_apply_relation_check.
+ */
parallel_apply_send_data(winfo, s->len, s->data);
elog(DEBUG1, "stopped streaming of xid %u, %u changes streamed",
@@ -2068,6 +2076,8 @@ apply_handle_insert(StringInfo s)
/* Set relation for error callback */
apply_error_callback_arg.rel = rel;
+ parallel_apply_relation_check(rel);
+
/* Initialize the executor state. */
edata = create_edata_for_relation(rel);
estate = edata->estate;
@@ -2211,6 +2221,8 @@ apply_handle_update(StringInfo s)
/* Check if we can do the update. */
check_relation_updatable(rel);
+ parallel_apply_relation_check(rel);
+
/* Initialize the executor state. */
edata = create_edata_for_relation(rel);
estate = edata->estate;
@@ -2379,6 +2391,8 @@ apply_handle_delete(StringInfo s)
/* Check if we can do the delete. */
check_relation_updatable(rel);
+ parallel_apply_relation_check(rel);
+
/* Initialize the executor state. */
edata = create_edata_for_relation(rel);
estate = edata->estate;
@@ -2564,13 +2578,14 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
+ part_entry = logicalrep_partition_open(relmapentry, partrel,
+ attrmap);
+
/* Check if we can do the update or delete on the leaf partition. */
if (operation == CMD_UPDATE || operation == CMD_DELETE)
- {
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
check_relation_updatable(part_entry);
- }
+
+ parallel_apply_relation_check(part_entry);
switch (operation)
{
diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c
index 808f9eb..b248899 100644
--- a/src/backend/utils/cache/typcache.c
+++ b/src/backend/utils/cache/typcache.c
@@ -2541,6 +2541,23 @@ compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2)
}
/*
+ * GetDomainConstraints --- get DomainConstraintState list of specified domain type
+ */
+List *
+GetDomainConstraints(Oid type_id)
+{
+ TypeCacheEntry *typentry;
+ List *constraints = NIL;
+
+ typentry = lookup_type_cache(type_id, TYPECACHE_DOMAIN_CONSTR_INFO);
+
+ if(typentry->domainData != NULL)
+ constraints = typentry->domainData->constraints;
+
+ return constraints;
+}
+
+/*
* Load (or re-load) the enumData member of the typcache entry.
*/
static void
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 64adf29..4d2b74b 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -113,6 +113,7 @@ typedef struct LogicalRepRelation
char replident; /* replica identity */
char relkind; /* remote relation kind */
Bitmapset *attkeys; /* Bitmap of key columns */
+ Bitmapset *attunique; /* Bitmap of unique columns */
} LogicalRepRelation;
/* Type mapping info */
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 78cd7e7..c778e7f 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -15,6 +15,17 @@
#include "access/attmap.h"
#include "replication/logicalproto.h"
+/*
+ * States to determine if changes on one relation can be applied using an
+ * parallel apply worker.
+ */
+typedef enum ParallelApplySafety
+{
+ PARALLEL_APPLY_SAFETY_UNKNOWN,
+ PARALLEL_APPLY_SAFE,
+ PARALLEL_APPLY_UNSAFE
+} ParallelApplySafety;
+
typedef struct LogicalRepRelMapEntry
{
LogicalRepRelation remoterel; /* key is remoterel.remoteid */
@@ -31,6 +42,8 @@ typedef struct LogicalRepRelMapEntry
Relation localrel; /* relcache entry (NULL when closed) */
AttrMap *attrmap; /* map of local attributes to remote ones */
bool updatable; /* Can apply updates/deletes? */
+ ParallelApplySafety parallel_apply_safe; /* Can apply changes in an
+ parallel apply worker? */
/* Sync state. */
char state;
@@ -47,4 +60,6 @@ extern LogicalRepRelMapEntry *logicalrep_partition_open(LogicalRepRelMapEntry *r
extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
LOCKMODE lockmode);
+extern void logicalrep_rel_mark_parallel_apply(LogicalRepRelMapEntry *entry);
+
#endif /* LOGICALRELATION_H */
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index fe8427b..191a3ae 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -217,6 +217,9 @@ extern void parallel_apply_replorigin_reset(void);
#define isParallelApplyWorker(worker) (worker->apply_leader_pid != InvalidPid)
+extern void parallel_apply_relation_check(LogicalRepRelMapEntry *rel);
+extern bool parallel_apply_using_worker(void);
+
static inline bool
am_tablesync_worker(void)
{
diff --git a/src/include/utils/typcache.h b/src/include/utils/typcache.h
index 431ad7f..ed7c2e7 100644
--- a/src/include/utils/typcache.h
+++ b/src/include/utils/typcache.h
@@ -199,6 +199,8 @@ extern uint64 assign_record_type_identifier(Oid type_id, int32 typmod);
extern int compare_values_of_enum(TypeCacheEntry *tcache, Oid arg1, Oid arg2);
+extern List *GetDomainConstraints(Oid type_id);
+
extern size_t SharedRecordTypmodRegistryEstimate(void);
extern void SharedRecordTypmodRegistryInit(SharedRecordTypmodRegistry *,
diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl
index 6844cc9..68eddce 100644
--- a/src/test/subscription/t/015_stream.pl
+++ b/src/test/subscription/t/015_stream.pl
@@ -206,8 +206,13 @@ my $oldpid = $node_publisher->safe_psql('postgres',
"SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
);
-$node_subscriber->safe_psql('postgres',
- "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel, binary = off)");
+# "streaming = parallel" does not support non-immutable functions, so change
+# the function in the default expression of column "c".
+$node_subscriber->safe_psql(
+ 'postgres', qq{
+ALTER TABLE test_tab ALTER COLUMN c SET DEFAULT to_timestamp(0);
+ALTER SUBSCRIPTION tap_sub SET(streaming = parallel, binary = off);
+});
$node_publisher->poll_query_until('postgres',
"SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl
index c5a10dc..5ec656e 100644
--- a/src/test/subscription/t/016_stream_subxact.pl
+++ b/src/test/subscription/t/016_stream_subxact.pl
@@ -151,8 +151,13 @@ my $oldpid = $node_publisher->safe_psql('postgres',
"SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
);
-$node_subscriber->safe_psql('postgres',
- "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)");
+# "streaming = parallel" does not support non-immutable functions, so change
+# the function in the default expression of column "c".
+$node_subscriber->safe_psql(
+ 'postgres', qq{
+ALTER TABLE test_tab ALTER COLUMN c SET DEFAULT to_timestamp(0);
+ALTER SUBSCRIPTION tap_sub SET(streaming = parallel);
+});
$node_publisher->poll_query_until('postgres',
"SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl
index 970fd85..b96e274 100644
--- a/src/test/subscription/t/022_twophase_cascade.pl
+++ b/src/test/subscription/t/022_twophase_cascade.pl
@@ -502,6 +502,14 @@ test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'on');
######################################
# Test using streaming mode 'parallel'
######################################
+
+# "streaming = parallel" does not support non-immutable functions, so change
+# the function in the default expression of column "c".
+$node_B->safe_psql(
+ 'postgres', "ALTER TABLE test_tab ALTER COLUMN c SET DEFAULT to_timestamp(0);");
+$node_C->safe_psql(
+ 'postgres', "ALTER TABLE test_tab ALTER COLUMN c SET DEFAULT to_timestamp(0);");
+
test_streaming($node_A, $node_B, $node_C, $appname_B, $appname_C, 'parallel');
###############################
diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl
index a457957..1fbc1c4 100644
--- a/src/test/subscription/t/023_twophase_stream.pl
+++ b/src/test/subscription/t/023_twophase_stream.pl
@@ -385,8 +385,13 @@ my $oldpid = $node_publisher->safe_psql('postgres',
"SELECT pid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
);
-$node_subscriber->safe_psql('postgres',
- "ALTER SUBSCRIPTION tap_sub SET(streaming = parallel)");
+# "streaming = parallel" does not support non-immutable functions, so change
+# the function in the default expression of column "c".
+$node_subscriber->safe_psql(
+ 'postgres', qq{
+ALTER TABLE test_tab ALTER COLUMN c SET DEFAULT to_timestamp(0);
+ALTER SUBSCRIPTION tap_sub SET(streaming = parallel);
+});
$node_publisher->poll_query_until('postgres',
"SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = '$appname' AND state = 'streaming';"
diff --git a/src/test/subscription/t/032_streaming_parallel_safety.pl b/src/test/subscription/t/032_streaming_parallel_safety.pl
new file mode 100644
index 0000000..24bd54f
--- /dev/null
+++ b/src/test/subscription/t/032_streaming_parallel_safety.pl
@@ -0,0 +1,608 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test the safety checks of streaming mode "parallel" in logical replication.
+# Without these safety checks, the subscriber's apply worker may fall into an
+# infinite wait without the user knowing.
+#
+# For normal tables, we use deadlock-producing test cases to ensure that future
+# modifications do not invalidate constraint checks.
+#
+# For partitioned tables, we just use test cases to confirm that the constraint
+# checks are as expected.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $offset = 0;
+
+# Create publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+# Setup structure on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab1 (a int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab2 (a int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab_partitioned (a int primary key, b varchar)");
+
+# Setup structure on subscriber
+# We need to test normal table and partition table.
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_tab1 (a int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_tab2 (a int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_tab_partitioned (a int primary key, b varchar) PARTITION BY RANGE(a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_tab_partition (LIKE test_tab_partitioned)");
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE test_tab_partitioned ATTACH PARTITION test_tab_partition DEFAULT"
+);
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_normal FOR TABLE test_tab1, test_tab2");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_partitioned FOR TABLE test_tab_partitioned");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql(
+ 'postgres', "
+ CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub_normal, tap_pub_partitioned
+ WITH (streaming = parallel, copy_data = false)");
+
+$node_publisher->wait_for_catchup($appname);
+
+# Interleave a pair of transactions, each exceeding the 64kB limit.
+my $in = '';
+my $out = '';
+
+my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
+
+my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
+ on_error_stop => 0);
+
+# ============================================================================
+# It is not allowed that the unique column in the relation on the
+# subscriber-side is not the unique column on the publisher-side. Check the
+# error reported by parallel worker in this case.
+# ============================================================================
+
+# First we check the unique index on normal table.
+$node_subscriber->safe_psql('postgres',
+ "CREATE UNIQUE INDEX idx_tab1 on test_tab1(a)");
+
+$in .= q{
+BEGIN;
+INSERT INTO test_tab1 SELECT i FROM generate_series(1, 5000) s(i);
+};
+$h->pump_nb;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO test_tab1 values(1)");
+
+$in .= q{
+COMMIT;
+\q
+};
+$h->finish;
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab1" using subscription parameter streaming = parallel/,
+ $offset);
+
+# Drop the unique index on the subscriber, now it works.
+$node_subscriber->safe_psql('postgres', "DROP INDEX idx_tab1");
+
+# Wait for this streaming transaction to be applied in the apply worker.
+$node_publisher->wait_for_catchup($appname);
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab1");
+is($result, qq(5001), 'data replicated to subscriber after dropping index');
+
+# Clean up test data from the environment.
+$node_publisher->safe_psql('postgres', "TRUNCATE TABLE test_tab1");
+$node_publisher->wait_for_catchup($appname);
+
+# Then we check the unique index on partition table.
+$node_subscriber->safe_psql('postgres',
+ "CREATE UNIQUE INDEX test_tab_b_partition_idx ON test_tab_partition (b)");
+
+# Check the subscriber log from now on.
+$offset = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)"
+);
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming = parallel/,
+ $offset);
+
+# Drop the unique index on the subscriber, now it works.
+$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx");
+
+# Wait for this streaming transaction to be applied in the apply worker.
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned");
+is($result, qq(5000), 'data replicated to subscriber after dropping index');
+
+# ============================================================================
+# Triggers which execute non-immutable function are not allowed on the
+# subscriber side. Check the error reported by parallel worker in this case.
+# ============================================================================
+
+# First we check the trigger function on normal table.
+$node_publisher->safe_psql('postgres',
+ "CREATE UNIQUE INDEX idx_tab2 on test_tab2(a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE UNIQUE INDEX idx_tab2 on test_tab2(a)");
+
+$node_subscriber->safe_psql(
+ 'postgres', qq{
+CREATE FUNCTION trigger_func_tab1_unsafe() RETURNS TRIGGER AS \$\$
+ BEGIN
+ INSERT INTO public.test_tab2 VALUES (NEW.*);
+ RETURN NEW;
+ END
+\$\$ language plpgsql;
+CREATE TRIGGER tri_tab1_unsafe
+BEFORE INSERT ON public.test_tab1
+FOR EACH ROW EXECUTE PROCEDURE trigger_func_tab1_unsafe();
+ALTER TABLE test_tab1 ENABLE REPLICA TRIGGER tri_tab1_unsafe;
+
+CREATE FUNCTION trigger_func_tab1_safe() RETURNS TRIGGER AS \$\$
+ BEGIN
+ RAISE NOTICE 'test for safe trigger function';
+ RETURN NEW;
+ END
+\$\$ language plpgsql;
+ALTER FUNCTION trigger_func_tab1_safe IMMUTABLE;
+CREATE TRIGGER tri_tab1_safe
+BEFORE INSERT ON public.test_tab1
+FOR EACH ROW EXECUTE PROCEDURE trigger_func_tab1_safe();
+});
+
+# Check the subscriber log from now on.
+$offset = -s $node_subscriber->logfile;
+
+$in .= q{
+BEGIN;
+INSERT INTO test_tab1 VALUES(5001);
+INSERT INTO test_tab2 SELECT i FROM generate_series(1, 5000) s(i);
+};
+$h->pump_nb;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO test_tab1 VALUES(5001)");
+
+$in .= q{
+COMMIT;
+\q
+};
+$h->finish;
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab1" using subscription parameter streaming = parallel/,
+ $offset);
+
+# Using trigger with immutable function, now it works.
+$node_subscriber->safe_psql(
+ 'postgres', qq{
+ALTER TABLE test_tab1 ENABLE REPLICA TRIGGER tri_tab1_safe;
+DROP TRIGGER tri_tab1_unsafe ON public.test_tab1;
+});
+
+# Wait for this streaming transaction to be applied in the apply worker.
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab1");
+is($result, qq(2), 'data replicated to subscriber after using immutable expression');
+
+# Clean up test data from the environment.
+$node_subscriber->safe_psql(
+ 'postgres', qq{
+DROP INDEX idx_tab2;
+DROP TRIGGER tri_tab1_safe ON public.test_tab1;
+DROP function trigger_func_tab1_unsafe;
+DROP function trigger_func_tab1_safe;
+});
+$node_publisher->safe_psql(
+ 'postgres', qq{
+DROP INDEX idx_tab2;
+TRUNCATE TABLE test_tab1;
+TRUNCATE TABLE test_tab2;
+});
+$node_publisher->wait_for_catchup($appname);
+
+# Then we check the trigger function on partition table.
+$node_subscriber->safe_psql(
+ 'postgres', qq{
+CREATE FUNCTION trigger_func() RETURNS TRIGGER AS \$\$
+ BEGIN
+ RETURN NULL;
+ END
+\$\$ language plpgsql;
+CREATE TRIGGER insert_trig
+BEFORE INSERT ON test_tab_partition
+FOR EACH ROW EXECUTE PROCEDURE trigger_func();
+ALTER TABLE test_tab_partition ENABLE REPLICA TRIGGER insert_trig;
+});
+
+# Check the subscriber log from now on.
+$offset = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming = parallel/,
+ $offset);
+
+# Drop the trigger on the subscriber, now it works.
+$node_subscriber->safe_psql('postgres',
+ "DROP TRIGGER insert_trig ON test_tab_partition");
+
+# Wait for this streaming transaction to be applied in the apply worker.
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned");
+is($result, qq(0), 'data replicated to subscriber after dropping trigger');
+
+# ============================================================================
+# It is not allowed that column default value expression contains a
+# non-immutable function on the subscriber side. Check the error reported by
+# parallel worker in this case.
+# ============================================================================
+
+# First we check the column default value expression on normal table.
+$node_publisher->safe_psql('postgres', "INSERT INTO test_tab2 VALUES(1)");
+
+$node_subscriber->safe_psql(
+ 'postgres', qq{
+CREATE FUNCTION func_count_tab2() RETURNS INT AS \$\$
+ BEGIN
+ RETURN (SELECT count(*) FROM public.test_tab2);
+ END
+\$\$ language plpgsql;
+ALTER TABLE test_tab1 ADD COLUMN b int DEFAULT func_count_tab2();
+});
+
+# Check the subscriber log from now on.
+$offset = -s $node_subscriber->logfile;
+
+$in .= q{
+BEGIN;
+TRUNCATE test_tab2;
+INSERT INTO test_tab1 SELECT i FROM generate_series(1, 5000) s(i);
+};
+$h->pump_nb;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO test_tab1(a) VALUES(1)");
+
+$in .= q{
+COMMIT;
+\q
+};
+$h->finish;
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab1" using subscription parameter streaming = parallel/,
+ $offset);
+
+# Alter default values to immutable expression, now it works.
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE test_tab1 ALTER COLUMN b SET DEFAULT 1");
+
+# Wait for this streaming transaction to be applied in the apply worker.
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab1");
+is($result, qq(5001),
+ 'data replicated to subscriber after using immutable expression');
+
+# Clean up test data from the environment.
+$node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab1 DROP COLUMN b");
+$node_publisher->safe_psql(
+ 'postgres', qq{
+TRUNCATE TABLE test_tab1;
+TRUNCATE TABLE test_tab2;
+});
+$node_publisher->wait_for_catchup($appname);
+
+# Then we check the column default value expression on partition table.
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE test_tab_partition ALTER COLUMN b SET DEFAULT random()");
+
+# Check the subscriber log from now on.
+$offset = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)"
+);
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming = parallel/,
+ $offset);
+
+# Drop default value on the subscriber, now it works.
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT");
+
+# Wait for this streaming transaction to be applied in the apply worker.
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned");
+is($result, qq(5000),
+ 'data replicated to subscriber after dropping default value expression');
+
+# ============================================================================
+# It is not allowed that domain constraint expression contains a non-immutable
+# function on the subscriber side. Check the error reported by parallel
+# worker in this case.
+# ============================================================================
+
+# Because the column type of the partition table must be the same as its parent
+# table, only test normal table here.
+$node_publisher->safe_psql('postgres', "INSERT INTO test_tab2 VALUES(1)");
+
+$node_publisher->safe_psql(
+ 'postgres', qq{
+CREATE DOMAIN tmp_domain AS int CHECK (VALUE > -1);
+ALTER TABLE test_tab1 ALTER COLUMN a TYPE tmp_domain;
+});
+
+$node_subscriber->safe_psql(
+ 'postgres', qq{
+CREATE DOMAIN tmp_domain AS INT CONSTRAINT domain_check CHECK (VALUE >= func_count_tab2());
+ALTER TABLE test_tab1 ALTER COLUMN a TYPE tmp_domain;
+});
+
+# Check the subscriber log from now on.
+$offset = -s $node_subscriber->logfile;
+
+$in .= q{
+BEGIN;
+TRUNCATE test_tab2;
+INSERT INTO test_tab1 SELECT i FROM generate_series(1, 5000) s(i);
+};
+$h->pump_nb;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO test_tab1(a) VALUES(1)");
+
+$in .= q{
+COMMIT;
+\q
+};
+$h->finish;
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab1" using subscription parameter streaming = parallel/,
+ $offset);
+
+# Drop domain constraint expression, now it works.
+$node_subscriber->safe_psql('postgres',
+ "ALTER DOMAIN tmp_domain DROP CONSTRAINT domain_check");
+
+# Wait for this streaming transaction to be applied in the apply worker.
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab1");
+is($result, qq(5001),
+ 'data replicated to subscriber after using immutable expression'
+);
+
+# Clean up test data from the environment.
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE test_tab1 ALTER COLUMN a TYPE int");
+$node_publisher->safe_psql(
+ 'postgres', qq{
+TRUNCATE TABLE test_tab1;
+TRUNCATE TABLE test_tab2;
+});
+$node_publisher->wait_for_catchup($appname);
+
+# ============================================================================
+# It is not allowed that constraint expression contains a non-immutable function
+# on the subscriber side. Check the error reported by parallel worker in this
+# case.
+# ============================================================================
+
+# First we check the constraint expression on normal table.
+$node_publisher->safe_psql('postgres', "INSERT INTO test_tab2 VALUES(1)");
+
+$node_subscriber->safe_psql(
+ 'postgres', qq{
+ALTER TABLE test_tab1 ADD CONSTRAINT const_tab1_unsafe CHECK(a >= func_count_tab2());
+});
+
+# Check the subscriber log from now on.
+$offset = -s $node_subscriber->logfile;
+
+$in .= q{
+BEGIN;
+TRUNCATE test_tab2;
+INSERT INTO test_tab1 SELECT i FROM generate_series(1, 5000) s(i);
+};
+$h->pump_nb;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO test_tab1(a) VALUES(1)");
+
+$in .= q{
+COMMIT;
+\q
+};
+$h->finish;
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab1" using subscription parameter streaming = parallel/,
+ $offset);
+
+# Alter constraint expression to immutable expression, now it works.
+$node_subscriber->safe_psql(
+ 'postgres', qq{
+ALTER TABLE test_tab1 DROP CONSTRAINT const_tab1_unsafe;
+ALTER TABLE test_tab1 ADD CONSTRAINT const_tab1_safe CHECK(a >= 0);
+});
+
+# Wait for this streaming transaction to be applied in the apply worker.
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab1");
+is($result, qq(5001),
+ 'data replicated to subscriber after using immutable expression');
+
+# Clean up test data from the environment.
+$node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab1 DROP CONSTRAINT const_tab1_safe");
+$node_publisher->safe_psql(
+ 'postgres', qq{
+TRUNCATE TABLE test_tab1;
+TRUNCATE TABLE test_tab2;
+});
+$node_publisher->wait_for_catchup($appname);
+
+# Then we check the constraint expression on partition table.
+$node_subscriber->safe_psql(
+ 'postgres', qq{
+ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_con check (a > random());
+});
+
+# Check the subscriber log from now on.
+$offset = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres', "DELETE FROM test_tab_partitioned");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming = parallel/,
+ $offset);
+
+# Drop constraint on the subscriber, now it works.
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con");
+
+# Wait for this streaming transaction to be applied in the apply worker.
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned");
+is($result, qq(0),
+ 'data replicated to subscriber after dropping constraint expression');
+
+# ============================================================================
+# It is not allowed that foreign key on the subscriber side. Check the error
+# reported by parallel worker in this case.
+# ============================================================================
+
+# First we check the foreign key on normal table.
+$node_publisher->safe_psql(
+ 'postgres', qq{
+CREATE TABLE tab_nopublic(a int);
+ALTER TABLE test_tab2 ADD PRIMARY KEY (a);
+ALTER TABLE test_tab2 REPLICA IDENTITY FULL;
+INSERT INTO test_tab2 VALUES(1);
+});
+$node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab2 ADD PRIMARY KEY (a);");
+
+$node_subscriber->safe_psql(
+ 'postgres', qq{
+ALTER TABLE test_tab1 ADD CONSTRAINT test_tab1fk FOREIGN KEY(a) REFERENCES test_tab2(a);
+SELECT 'ALTER TABLE test_tab1 ENABLE REPLICA TRIGGER "' || tgname || '"' FROM pg_trigger WHERE tgrelid = 'test_tab1'::regclass::oid \\gexec
+});
+
+# Check the subscriber log from now on.
+$offset = -s $node_subscriber->logfile;
+
+$in .= q{
+BEGIN;
+INSERT INTO test_tab1(a) VALUES(1);
+INSERT INTO tab_nopublic SELECT i FROM generate_series(1, 5000) s(i);
+};
+$h->pump_nb;
+
+$node_publisher->safe_psql('postgres', "DELETE FROM test_tab2");
+
+$in .= q{
+COMMIT;
+\q
+};
+$h->finish;
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab1" using subscription parameter streaming = parallel/,
+ $offset);
+
+# Drop the foreign key constraint on the subscriber, now it works.
+$node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab1 DROP CONSTRAINT test_tab1fk");
+
+# Wait for this streaming transaction to be applied in the apply worker.
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab1");
+is($result, qq(1),
+ 'data replicated to subscriber after dropping the foreign key');
+
+# Clean up test data from the environment.
+$node_publisher->safe_psql(
+ 'postgres', qq{
+TRUNCATE TABLE test_tab1;
+TRUNCATE TABLE test_tab2;
+});
+$node_publisher->wait_for_catchup($appname);
+
+# Then we check the foreign key on partition table.
+$node_subscriber->safe_psql(
+ 'postgres', qq{
+CREATE TABLE test_tab_partition_f (a int primary key);
+ALTER TABLE test_tab_partition ADD CONSTRAINT test_tab_patition_fk FOREIGN KEY(a) REFERENCES test_tab_partition_f(a);
+});
+
+# Check the subscriber log from now on.
+$offset = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab_partitioned SELECT i, md5(i::text) FROM generate_series(1, 5000) s(i)"
+);
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming = parallel/,
+ $offset);
+
+# Drop the foreign key constraint on the subscriber, now it works.
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk");
+
+# Wait for this streaming transaction to be applied in the apply worker.
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned");
+is($result, qq(5000),
+ 'data replicated to subscriber after dropping the foreign key');
+
+$node_subscriber->stop;
+$node_publisher->stop;
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b5a4fde..99491fb 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1892,6 +1892,7 @@ PageXLogRecPtr
PagetableEntry
Pairs
ParallelAppendState
+ParallelApplySafety
ParallelBitmapHeapState
ParallelBlockTableScanDesc
ParallelBlockTableScanWorker
--
2.7.2.windows.1
[application/octet-stream] v26-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch (60.1K, ../OS0PR01MB5716546863340CA19EC5F8C394789@OS0PR01MB5716.jpnprd01.prod.outlook.com/6-v26-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch)
download | inline diff:
From d71b8b61ddc399379b041997acd4ccb3cfff93f4 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <[email protected]>
Date: Wed, 24 Aug 2022 21:07:44 +0800
Subject: [PATCH v26 4/5] Retry to apply streaming xact only in apply worker
When the subscription parameter is set streaming=parallel, the logic tries to
apply the streaming transaction using an parallel apply worker. If this
fails the parallel worker exits with an error.
In this case, retry applying the streaming transaction using the normal
streaming=on mode. This is done to avoid getting caught in a loop of the same
retry errors.
A new flag field "subretry" has been introduced to catalog "pg_subscription".
If the subscriber exits with an error, this flag will be set true, and
whenever the transaction is applied successfully, this flag is reset false.
Now, when deciding how to apply a streaming transaction, the logic can know if
this transaction has previously failed or not (by checking the "subretry"
field).
In addition, when introduced parallel apply worker, we added some log checks to
ensure that streamed transactions are applied in parallel apply worker.
But since we introduced "retry" now, we will not be able to confirm whether
streamed transactions will be applied in parallel apply worker. So removed
these log checks.
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/ref/create_subscription.sgml | 5 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 1 +
.../replication/logical/applyparallelworker.c | 16 +-
src/backend/replication/logical/worker.c | 187 +++++++++++++++------
src/bin/pg_dump/pg_dump.c | 5 +-
src/include/catalog/pg_subscription.h | 4 +
src/test/subscription/t/015_stream.pl | 57 +------
src/test/subscription/t/016_stream_subxact.pl | 46 +----
src/test/subscription/t/017_stream_ddl.pl | 52 +-----
.../subscription/t/018_stream_subxact_abort.pl | 57 +------
.../subscription/t/019_stream_subxact_ddl_abort.pl | 47 +-----
src/test/subscription/t/022_twophase_cascade.pl | 55 ------
src/test/subscription/t/023_twophase_stream.pl | 67 +-------
.../t/032_streaming_parallel_safety.pl | 134 +++++++--------
17 files changed, 254 insertions(+), 491 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index e5e8b4d..97889e8 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7929,6 +7929,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<row>
<entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subretry</structfield> <type>bool</type>
+ </para>
+ <para>
+ True if the previous apply change failed, necessitating a retry
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
</para>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 325eee8..17c2f58 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -246,6 +246,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
any non-immutable functions used by the subscriber-side replicated
table. 3) there cannot be any foreign keys used by the
subscriber-side replicated table.
+ When applying a streaming transaction, if either requirement is not
+ met, the parallel apply worker will exit with an error.
+ The <literal>parallel</literal> mode is disregarded when retrying;
+ instead the transaction will be applied using <literal>on</literal>
+ mode.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index a506fc3..723b141 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->stream = subform->substream;
sub->twophasestate = subform->subtwophasestate;
sub->disableonerr = subform->subdisableonerr;
+ sub->retry = subform->subretry;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5a844b6..652d76f 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1299,7 +1299,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
- subslotname, subsynccommit, subpublications, suborigin)
+ subretry, subslotname, subsynccommit, subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 3dc03dc..ac92cbb 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -689,6 +689,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
LOGICALREP_TWOPHASE_STATE_PENDING :
LOGICALREP_TWOPHASE_STATE_DISABLED);
values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
+ values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(false);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index dea19bb..0a675e5 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -138,6 +138,18 @@ parallel_apply_can_start(TransactionId xid)
return false;
/*
+ * Don't use apply apply workers for retries, because it is possible
+ * that the last time we tried to apply a transaction using an apply
+ * apply worker the checks failed (see function
+ * parallel_apply_relation_check).
+ */
+ if (MySubscription->retry)
+ {
+ elog(DEBUG1, "parallel apply workers are not used for retries");
+ return false;
+ }
+
+ /*
* For streaming transactions that are being applied using parallel apply
* worker, we cannot decide whether to apply the change for a relation that
* is not in the READY state (see should_apply_changes_for_rel) as we won't
@@ -1158,7 +1170,5 @@ parallel_apply_relation_check(LogicalRepRelMapEntry *rel)
"streaming = parallel"),
errdetail("The unique column on subscriber is not the unique "
"column on publisher or there is at least one "
- "non-immutable function."),
- errhint("Please change to use subscription parameter %s.",
- "streaming = on")));
+ "non-immutable function.")));
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a7e1ea2..5edb0fa 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -388,7 +388,7 @@ static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
-static void DisableSubscriptionAndExit(void);
+static void DisableSubscriptionOnError(void);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
@@ -428,6 +428,8 @@ static inline void reset_apply_error_context_info(void);
static TransactionApplyAction get_transaction_apply_action(TransactionId xid,
ParallelApplyWorkerInfo **winfo);
+static void set_subscription_retry(bool retry);
+
/*
* Should this worker apply changes for given relation.
*
@@ -980,6 +982,9 @@ apply_handle_commit(StringInfo s)
apply_handle_commit_internal(&commit_data);
+ /* Reset the retry flag. */
+ set_subscription_retry(false);
+
/* Process any tables that are being synchronized in parallel. */
process_syncing_tables(commit_data.end_lsn);
@@ -1089,6 +1094,9 @@ apply_handle_prepare(StringInfo s)
in_remote_transaction = false;
+ /* Reset the retry flag. */
+ set_subscription_retry(false);
+
/* Process any tables that are being synchronized in parallel. */
process_syncing_tables(prepare_data.end_lsn);
@@ -1145,6 +1153,9 @@ apply_handle_commit_prepared(StringInfo s)
store_flush_position(prepare_data.end_lsn);
in_remote_transaction = false;
+ /* Reset the retry flag. */
+ set_subscription_retry(false);
+
/* Process any tables that are being synchronized in parallel. */
process_syncing_tables(prepare_data.end_lsn);
@@ -1206,6 +1217,9 @@ apply_handle_rollback_prepared(StringInfo s)
store_flush_position(rollback_data.rollback_end_lsn);
in_remote_transaction = false;
+ /* Reset the retry flag. */
+ set_subscription_retry(false);
+
/* Process any tables that are being synchronized in parallel. */
process_syncing_tables(rollback_data.rollback_end_lsn);
@@ -1278,6 +1292,9 @@ apply_handle_stream_prepare(StringInfo s)
prepare_data.xid);
store_flush_position(prepare_data.end_lsn);
+
+ /* Reset the retry flag. */
+ set_subscription_retry(false);
}
else if (apply_action == TA_SERIALIZE_TO_FILE)
{
@@ -1296,6 +1313,9 @@ apply_handle_stream_prepare(StringInfo s)
/* Unlink the files with serialized changes and subxact info. */
stream_cleanup_files(MyLogicalRepWorker->subid, prepare_data.xid);
+
+ /* Reset the retry flag. */
+ set_subscription_retry(false);
}
else
Assert(false);
@@ -1662,6 +1682,9 @@ apply_handle_stream_abort(StringInfo s)
elog(DEBUG1, "streamed transaction %u was aborted using an parallel apply worker",
xid);
+
+ /* Reset the retry flag. */
+ set_subscription_retry(false);
}
}
else if (apply_action == TA_SERIALIZE_TO_FILE)
@@ -1671,6 +1694,10 @@ apply_handle_stream_abort(StringInfo s)
* serialized to file.
*/
serialize_stream_abort(xid, subxid);
+
+ /* Reset the retry flag. */
+ if (subxid == xid)
+ set_subscription_retry(false);
}
else
Assert(false);
@@ -1863,6 +1890,9 @@ apply_handle_stream_commit(StringInfo s)
* subskiplsn.
*/
clear_subscription_skip_lsn(commit_data.commit_lsn);
+
+ /* Reset the retry flag. */
+ set_subscription_retry(false);
}
else if (apply_action == TA_SERIALIZE_TO_FILE)
{
@@ -1876,6 +1906,9 @@ apply_handle_stream_commit(StringInfo s)
/* Unlink the files with serialized changes and subxact info. */
stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ /* Reset the retry flag. */
+ set_subscription_retry(false);
}
else
Assert(false);
@@ -3925,20 +3958,28 @@ start_table_sync(XLogRecPtr *origin_startpos, char **myslotname)
}
PG_CATCH();
{
+ /*
+ * Emit the error message, and recover from the error state to an idle
+ * state
+ */
+ HOLD_INTERRUPTS();
+
+ EmitErrorReport();
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+
+ RESUME_INTERRUPTS();
+
+ /* Report the worker failed during table synchronization */
+ pgstat_report_subscription_error(MySubscription->oid, false);
+
if (MySubscription->disableonerr)
- DisableSubscriptionAndExit();
- else
- {
- /*
- * Report the worker failed during table synchronization. Abort
- * the current transaction so that the stats message is sent in an
- * idle state.
- */
- AbortOutOfAnyTransaction();
- pgstat_report_subscription_error(MySubscription->oid, false);
+ DisableSubscriptionOnError();
- PG_RE_THROW();
- }
+ /* Set the retry flag. */
+ set_subscription_retry(true);
+
+ proc_exit(0);
}
PG_END_TRY();
@@ -3963,20 +4004,27 @@ start_apply(XLogRecPtr origin_startpos)
}
PG_CATCH();
{
+ /*
+ * Emit the error message, and recover from the error state to an idle
+ * state
+ */
+ HOLD_INTERRUPTS();
+
+ EmitErrorReport();
+ AbortOutOfAnyTransaction();
+ FlushErrorState();
+
+ RESUME_INTERRUPTS();
+
+ /* Report the worker failed while applying changes */
+ pgstat_report_subscription_error(MySubscription->oid,
+ !am_tablesync_worker());
+
if (MySubscription->disableonerr)
- DisableSubscriptionAndExit();
- else
- {
- /*
- * Report the worker failed while applying changes. Abort the
- * current transaction so that the stats message is sent in an
- * idle state.
- */
- AbortOutOfAnyTransaction();
- pgstat_report_subscription_error(MySubscription->oid, !am_tablesync_worker());
+ DisableSubscriptionOnError();
- PG_RE_THROW();
- }
+ /* Set the retry flag. */
+ set_subscription_retry(true);
}
PG_END_TRY();
}
@@ -4238,39 +4286,20 @@ ApplyWorkerMain(Datum main_arg)
}
/*
- * After error recovery, disable the subscription in a new transaction
- * and exit cleanly.
+ * Disable the subscription in a new transaction.
*/
static void
-DisableSubscriptionAndExit(void)
+DisableSubscriptionOnError(void)
{
- /*
- * Emit the error message, and recover from the error state to an idle
- * state
- */
- HOLD_INTERRUPTS();
-
- EmitErrorReport();
- AbortOutOfAnyTransaction();
- FlushErrorState();
-
- RESUME_INTERRUPTS();
-
- /* Report the worker failed during either table synchronization or apply */
- pgstat_report_subscription_error(MyLogicalRepWorker->subid,
- !am_tablesync_worker());
-
/* Disable the subscription */
StartTransactionCommand();
DisableSubscription(MySubscription->oid);
CommitTransactionCommand();
- /* Notify the subscription has been disabled and exit */
+ /* Notify the subscription has been disabled */
ereport(LOG,
errmsg("logical replication subscription \"%s\" has been disabled due to an error",
MySubscription->name));
-
- proc_exit(0);
}
/*
@@ -4537,3 +4566,67 @@ get_transaction_apply_action(TransactionId xid, ParallelApplyWorkerInfo **winfo)
else
return TA_SERIALIZE_TO_FILE;
}
+
+/*
+ * Set subretry of pg_subscription catalog.
+ *
+ * If retry is true, subscriber is about to exit with an error. Otherwise, it
+ * means that the transaction was applied successfully.
+ */
+static void
+set_subscription_retry(bool retry)
+{
+ Relation rel;
+ HeapTuple tup;
+ bool started_tx = false;
+ bool nulls[Natts_pg_subscription];
+ bool replaces[Natts_pg_subscription];
+ Datum values[Natts_pg_subscription];
+
+ /* Fast path - if no state change then nothing to do */
+ if (MySubscription->retry == retry)
+ return;
+
+ /* Fast path - skip for parallel apply workers */
+ if (am_parallel_apply_worker())
+ return;
+
+ if (!IsTransactionState())
+ {
+ StartTransactionCommand();
+ started_tx = true;
+ }
+
+ /* Look up the subscription in the catalog */
+ rel = table_open(SubscriptionRelationId, RowExclusiveLock);
+ tup = SearchSysCacheCopy1(SUBSCRIPTIONOID,
+ ObjectIdGetDatum(MySubscription->oid));
+
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "subscription \"%s\" does not exist", MySubscription->name);
+
+ LockSharedObject(SubscriptionRelationId, MySubscription->oid, 0,
+ AccessShareLock);
+
+ /* Form a new tuple. */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ /* Reset subretry */
+ values[Anum_pg_subscription_subretry - 1] = BoolGetDatum(retry);
+ replaces[Anum_pg_subscription_subretry - 1] = true;
+
+ tup = heap_modify_tuple(tup, RelationGetDescr(rel), values, nulls,
+ replaces);
+
+ /* Update the catalog. */
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+
+ /* Cleanup. */
+ heap_freetuple(tup);
+ table_close(rel, NoLock);
+
+ if (started_tx)
+ CommitTransactionCommand();
+}
\ No newline at end of file
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 13c8e10..81d566d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4507,8 +4507,9 @@ getSubscriptions(Archive *fout)
ntups = PQntuples(res);
/*
- * Get subscription fields. We don't include subskiplsn in the dump as
- * after restoring the dump this value may no longer be relevant.
+ * Get subscription fields. We don't include subskiplsn and subretry in
+ * the dump as after restoring the dump this value may no longer be
+ * relevant.
*/
i_tableoid = PQfnumber(res, "tableoid");
i_oid = PQfnumber(res, "oid");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 79ff745..e2516b5 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -88,6 +88,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subdisableonerr; /* True if a worker error should cause the
* subscription to be disabled */
+ bool subretry BKI_DEFAULT(f); /* True if the previous apply change failed. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -131,6 +133,8 @@ typedef struct Subscription
bool disableonerr; /* Indicates if the subscription should be
* automatically disabled if a worker error
* occurs */
+ bool retry; /* Indicates if the previous apply change
+ * failed. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl
index 68eddce..742bce0 100644
--- a/src/test/subscription/t/015_stream.pl
+++ b/src/test/subscription/t/015_stream.pl
@@ -8,53 +8,22 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
-# Check the log that the streamed transaction was completed successfully
-# reported by parallel apply worker.
-sub check_parallel_log
-{
- my ($node_subscriber, $offset, $is_parallel, $action)= @_;
- my $parallel_message = '';
-
- if ($is_parallel)
- {
- $parallel_message = join("", 'streamed transaction \d+ was ', $action, ' using an parallel apply worker');
-
- $node_subscriber->wait_for_log(qr/$parallel_message/, $offset);
- }
-}
-
# Encapsulate all the common test steps which are related to "streaming"
# parameter so the same code can be run both for the streaming=on and
# streaming=parallel cases.
sub test_streaming
{
- my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
+ my ($node_publisher, $node_subscriber, $appname) = @_;
# Interleave a pair of transactions, each exceeding the 64kB limit.
my $in = '';
my $out = '';
- my $offset = 0;
-
my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
on_error_stop => 0);
- # If "streaming" parameter is specified as "parallel", we need to check
- # that streamed transaction was applied using an parallel apply worker.
- # We have to look for the DEBUG1 log messages about that, so bump up the
- # log verbosity.
- if ($is_parallel)
- {
- $node_subscriber->append_conf('postgresql.conf',
- "log_min_messages = debug1");
- $node_subscriber->reload;
- }
-
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
-
$in .= q{
BEGIN;
INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
@@ -79,8 +48,6 @@ sub test_streaming
$node_publisher->wait_for_catchup($appname);
- check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
-
my $result =
$node_subscriber->safe_psql('postgres',
"SELECT count(*), count(c), count(d = 999) FROM test_tab");
@@ -91,9 +58,6 @@ sub test_streaming
$node_subscriber->safe_psql('postgres',
"ALTER SUBSCRIPTION tap_sub SET (binary = on)");
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
-
# Insert, update and delete enough rows to exceed the 64kB limit.
$node_publisher->safe_psql(
'postgres', q{
@@ -106,8 +70,6 @@ sub test_streaming
$node_publisher->wait_for_catchup($appname);
- check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
-
$result =
$node_subscriber->safe_psql('postgres',
"SELECT count(*), count(c), count(d = 999) FROM test_tab");
@@ -122,16 +84,11 @@ sub test_streaming
"UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'"
);
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
-
$node_publisher->safe_psql('postgres',
"UPDATE test_tab SET b = md5(a::text)");
$node_publisher->wait_for_catchup($appname);
- check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
-
$result = $node_subscriber->safe_psql('postgres',
"SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab"
);
@@ -142,14 +99,6 @@ sub test_streaming
$node_publisher->safe_psql('postgres',
"DELETE FROM test_tab WHERE (a > 2)");
$node_publisher->wait_for_catchup($appname);
-
- # Reset the log verbosity.
- if ($is_parallel)
- {
- $node_subscriber->append_conf('postgresql.conf',
- "log_min_messages = warning");
- $node_subscriber->reload;
- }
}
# Create publisher node
@@ -197,7 +146,7 @@ my $result =
"SELECT count(*), count(c), count(d = 999) FROM test_tab");
is($result, qq(2|2|2), 'check initial data was copied to subscriber');
-test_streaming($node_publisher, $node_subscriber, $appname, 0);
+test_streaming($node_publisher, $node_subscriber, $appname);
######################################
# Test using streaming mode 'parallel'
@@ -220,7 +169,7 @@ $node_publisher->poll_query_until('postgres',
or die
"Timed out while waiting for apply to restart after changing SUBSCRIPTION";
-test_streaming($node_publisher, $node_subscriber, $appname, 1);
+test_streaming($node_publisher, $node_subscriber, $appname);
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/016_stream_subxact.pl b/src/test/subscription/t/016_stream_subxact.pl
index 5ec656e..ab22f92 100644
--- a/src/test/subscription/t/016_stream_subxact.pl
+++ b/src/test/subscription/t/016_stream_subxact.pl
@@ -8,43 +8,12 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
-# Check the log that the streamed transaction was completed successfully
-# reported by parallel apply worker.
-sub check_parallel_log
-{
- my ($node_subscriber, $offset, $is_parallel, $action)= @_;
- my $parallel_message = '';
-
- if ($is_parallel)
- {
- $parallel_message = join("", 'streamed transaction \d+ was ', $action, ' using an parallel apply worker');
-
- $node_subscriber->wait_for_log(qr/$parallel_message/, $offset);
- }
-}
-
# Encapsulate all the common test steps which are related to "streaming"
# parameter so the same code can be run both for the streaming=on and
# streaming=parallel cases.
sub test_streaming
{
- my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
-
- my $offset = 0;
-
- # If "streaming" parameter is specified as "parallel", we need to check
- # that streamed transaction was applied using an parallel apply worker.
- # We have to look for the DEBUG1 log messages about that, so bump up the
- # log verbosity.
- if ($is_parallel)
- {
- $node_subscriber->append_conf('postgresql.conf',
- "log_min_messages = debug1");
- $node_subscriber->reload;
- }
-
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
+ my ($node_publisher, $node_subscriber, $appname) = @_;
# Insert, update and delete enough rows to exceed 64kB limit.
$node_publisher->safe_psql(
@@ -74,8 +43,6 @@ sub test_streaming
$node_publisher->wait_for_catchup($appname);
- check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
-
my $result =
$node_subscriber->safe_psql('postgres',
"SELECT count(*), count(c), count(d = 999) FROM test_tab");
@@ -88,13 +55,6 @@ sub test_streaming
"DELETE FROM test_tab WHERE (a > 2)");
$node_publisher->wait_for_catchup($appname);
- # Reset the log verbosity.
- if ($is_parallel)
- {
- $node_subscriber->append_conf('postgresql.conf',
- "log_min_messages = warning");
- $node_subscriber->reload;
- }
}
# Create publisher node
@@ -142,7 +102,7 @@ my $result =
"SELECT count(*), count(c), count(d = 999) FROM test_tab");
is($result, qq(2|2|2), 'check initial data was copied to subscriber');
-test_streaming($node_publisher, $node_subscriber, $appname, 0);
+test_streaming($node_publisher, $node_subscriber, $appname);
######################################
# Test using streaming mode 'parallel'
@@ -165,7 +125,7 @@ $node_publisher->poll_query_until('postgres',
or die
"Timed out while waiting for apply to restart after changing SUBSCRIPTION";
-test_streaming($node_publisher, $node_subscriber, $appname, 1);
+test_streaming($node_publisher, $node_subscriber, $appname);
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/017_stream_ddl.pl b/src/test/subscription/t/017_stream_ddl.pl
index 0beca11..43f9ca3 100644
--- a/src/test/subscription/t/017_stream_ddl.pl
+++ b/src/test/subscription/t/017_stream_ddl.pl
@@ -8,29 +8,12 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
-# Check the log that the streamed transaction was completed successfully
-# reported by parallel apply worker.
-sub check_parallel_log
-{
- my ($node_subscriber, $offset, $is_parallel, $action)= @_;
- my $parallel_message = '';
-
- if ($is_parallel)
- {
- $parallel_message = join("", 'streamed transaction \d+ was ', $action, ' using an parallel apply worker');
-
- $node_subscriber->wait_for_log(qr/$parallel_message/, $offset);
- }
-}
-
# Encapsulate all the common test steps which are related to "streaming"
# parameter so the same code can be run both for the streaming=on and
# streaming=parallel cases.
sub test_streaming
{
- my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
-
- my $offset = 0;
+ my ($node_publisher, $node_subscriber, $appname) = @_;
# a small (non-streamed) transaction with DDL and DML
$node_publisher->safe_psql(
@@ -43,20 +26,6 @@ sub test_streaming
COMMIT;
});
- # If "streaming" parameter is specified as "parallel", we need to check
- # that streamed transaction was applied using an parallel apply worker.
- # We have to look for the DEBUG1 log messages about that, so bump up the
- # log verbosity.
- if ($is_parallel)
- {
- $node_subscriber->append_conf('postgresql.conf',
- "log_min_messages = debug1");
- $node_subscriber->reload;
- }
-
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
-
# large (streamed) transaction with DDL and DML
$node_publisher->safe_psql(
'postgres', q{
@@ -81,8 +50,6 @@ sub test_streaming
$node_publisher->wait_for_catchup($appname);
- check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
-
my $result =
$node_subscriber->safe_psql('postgres',
"SELECT count(*), count(c), count(d), count(e) FROM test_tab");
@@ -90,9 +57,6 @@ sub test_streaming
'check data was copied to subscriber in streaming mode and extra columns contain local defaults'
);
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
-
# A large (streamed) transaction with DDL and DML. One of the DDL is performed
# after DML to ensure that we invalidate the schema sent for test_tab so that
# the next transaction has to send the schema again.
@@ -115,8 +79,6 @@ sub test_streaming
$node_publisher->wait_for_catchup($appname);
- check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
-
$result = $node_subscriber->safe_psql('postgres',
"SELECT count(*), count(c), count(d), count(e), count(f) FROM test_tab"
);
@@ -131,14 +93,6 @@ sub test_streaming
ALTER TABLE test_tab DROP COLUMN c, DROP COLUMN d, DROP COLUMN e, DROP COLUMN f;
});
$node_publisher->wait_for_catchup($appname);
-
- # Reset the log verbosity.
- if ($is_parallel)
- {
- $node_subscriber->append_conf('postgresql.conf',
- "log_min_messages = warning");
- $node_subscriber->reload;
- }
}
# Create publisher node
@@ -186,7 +140,7 @@ my $result =
"SELECT count(*), count(c), count(d = 999) FROM test_tab");
is($result, qq(2|0|0), 'check initial data was copied to subscriber');
-test_streaming($node_publisher, $node_subscriber, $appname, 0);
+test_streaming($node_publisher, $node_subscriber, $appname);
######################################
# Test using streaming mode 'parallel'
@@ -204,7 +158,7 @@ $node_publisher->poll_query_until('postgres',
or die
"Timed out while waiting for apply to restart after changing SUBSCRIPTION";
-test_streaming($node_publisher, $node_subscriber, $appname, 1);
+test_streaming($node_publisher, $node_subscriber, $appname);
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/018_stream_subxact_abort.pl b/src/test/subscription/t/018_stream_subxact_abort.pl
index 12f693b..67c778d 100644
--- a/src/test/subscription/t/018_stream_subxact_abort.pl
+++ b/src/test/subscription/t/018_stream_subxact_abort.pl
@@ -8,43 +8,12 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
-# Check the log that the streamed transaction was completed successfully
-# reported by parallel apply worker.
-sub check_parallel_log
-{
- my ($node_subscriber, $offset, $is_parallel, $action)= @_;
- my $parallel_message = '';
-
- if ($is_parallel)
- {
- $parallel_message = join("", 'streamed transaction \d+ was ', $action, ' using an parallel apply worker');
-
- $node_subscriber->wait_for_log(qr/$parallel_message/, $offset);
- }
-}
-
# Encapsulate all the common test steps which are related to "streaming"
# parameter so the same code can be run both for the streaming=on and
# streaming=parallel cases.
sub test_streaming
{
- my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
-
- my $offset = 0;
-
- # If "streaming" parameter is specified as "parallel", we need to check
- # that streamed transaction was applied using an parallel apply worker.
- # We have to look for the DEBUG1 log messages about that, so bump up the
- # log verbosity.
- if ($is_parallel)
- {
- $node_subscriber->append_conf('postgresql.conf',
- "log_min_messages = debug1");
- $node_subscriber->reload;
- }
-
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
+ my ($node_publisher, $node_subscriber, $appname) = @_;
# large (streamed) transaction with DDL, DML and ROLLBACKs
$node_publisher->safe_psql(
@@ -70,8 +39,6 @@ sub test_streaming
$node_publisher->wait_for_catchup($appname);
- check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
-
my $result =
$node_subscriber->safe_psql('postgres',
"SELECT count(*), count(c) FROM test_tab");
@@ -79,9 +46,6 @@ sub test_streaming
'check rollback to savepoint was reflected on subscriber and extra columns contain local defaults'
);
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
-
# large (streamed) transaction with subscriber receiving out of order
# subtransaction ROLLBACKs
$node_publisher->safe_psql(
@@ -102,17 +66,12 @@ sub test_streaming
$node_publisher->wait_for_catchup($appname);
- check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
-
$result =
$node_subscriber->safe_psql('postgres',
"SELECT count(*), count(c) FROM test_tab");
is($result, qq(2500|0),
'check rollback to savepoint was reflected on subscriber');
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
-
# large (streamed) transaction with subscriber receiving rollback
$node_publisher->safe_psql(
'postgres', q{
@@ -127,8 +86,6 @@ sub test_streaming
$node_publisher->wait_for_catchup($appname);
- check_parallel_log($node_subscriber, $offset, $is_parallel, "aborted");
-
$result =
$node_subscriber->safe_psql('postgres',
"SELECT count(*), count(c) FROM test_tab");
@@ -138,14 +95,6 @@ sub test_streaming
$node_publisher->safe_psql('postgres',
"DELETE FROM test_tab WHERE (a > 2)");
$node_publisher->wait_for_catchup($appname);
-
- # Reset the log verbosity.
- if ($is_parallel)
- {
- $node_subscriber->append_conf('postgresql.conf',
- "log_min_messages = warning");
- $node_subscriber->reload;
- }
}
# Create publisher node
@@ -192,7 +141,7 @@ my $result =
"SELECT count(*), count(c) FROM test_tab");
is($result, qq(2|0), 'check initial data was copied to subscriber');
-test_streaming($node_publisher, $node_subscriber, $appname, 0);
+test_streaming($node_publisher, $node_subscriber, $appname);
######################################
# Test using streaming mode 'parallel'
@@ -210,7 +159,7 @@ $node_publisher->poll_query_until('postgres',
or die
"Timed out while waiting for apply to restart after changing SUBSCRIPTION";
-test_streaming($node_publisher, $node_subscriber, $appname, 1);
+test_streaming($node_publisher, $node_subscriber, $appname);
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl
index 90a77e5..41ec1e8 100644
--- a/src/test/subscription/t/019_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/019_stream_subxact_ddl_abort.pl
@@ -9,43 +9,12 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
-# Check the log that the streamed transaction was completed successfully
-# reported by parallel apply worker.
-sub check_parallel_log
-{
- my ($node_subscriber, $offset, $is_parallel, $action)= @_;
- my $parallel_message = '';
-
- if ($is_parallel)
- {
- $parallel_message = join("", 'streamed transaction \d+ was ', $action, ' using an parallel apply worker');
-
- $node_subscriber->wait_for_log(qr/$parallel_message/, $offset);
- }
-}
-
# Encapsulate all the common test steps which are related to "streaming"
# parameter so the same code can be run both for the streaming=on and
# streaming=parallel cases.
sub test_streaming
{
- my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
-
- my $offset = 0;
-
- # If "streaming" parameter is specified as "parallel", we need to check
- # that streamed transaction was applied using an parallel apply worker.
- # We have to look for the DEBUG1 log messages about that, so bump up the
- # log verbosity.
- if ($is_parallel)
- {
- $node_subscriber->append_conf('postgresql.conf',
- "log_min_messages = debug1");
- $node_subscriber->reload;
- }
-
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
+ my ($node_publisher, $node_subscriber, $appname) = @_;
# large (streamed) transaction with DDL, DML and ROLLBACKs
$node_publisher->safe_psql(
@@ -69,8 +38,6 @@ sub test_streaming
$node_publisher->wait_for_catchup($appname);
- check_parallel_log($node_subscriber, $offset, $is_parallel, "committed");
-
my $result =
$node_subscriber->safe_psql('postgres',
"SELECT count(*), count(c) FROM test_tab");
@@ -85,14 +52,6 @@ sub test_streaming
ALTER TABLE test_tab DROP COLUMN c;
});
$node_publisher->wait_for_catchup($appname);
-
- # Reset the log verbosity.
- if ($is_parallel)
- {
- $node_subscriber->append_conf('postgresql.conf',
- "log_min_messages = warning");
- $node_subscriber->reload;
- }
}
# Create publisher node
@@ -139,7 +98,7 @@ my $result =
"SELECT count(*), count(c) FROM test_tab");
is($result, qq(2|0), 'check initial data was copied to subscriber');
-test_streaming($node_publisher, $node_subscriber, $appname, 0);
+test_streaming($node_publisher, $node_subscriber, $appname);
######################################
# Test using streaming mode 'parallel'
@@ -157,7 +116,7 @@ $node_publisher->poll_query_until('postgres',
or die
"Timed out while waiting for apply to restart after changing SUBSCRIPTION";
-test_streaming($node_publisher, $node_subscriber, $appname, 1);
+test_streaming($node_publisher, $node_subscriber, $appname);
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/022_twophase_cascade.pl b/src/test/subscription/t/022_twophase_cascade.pl
index b96e274..e8f2073 100644
--- a/src/test/subscription/t/022_twophase_cascade.pl
+++ b/src/test/subscription/t/022_twophase_cascade.pl
@@ -11,21 +11,6 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
-# Check the log that the streamed transaction was completed successfully
-# reported by parallel apply worker.
-sub check_parallel_log
-{
- my ($node_subscriber, $offset, $streaming_mode, $action)= @_;
- my $parallel_message = '';
-
- if ($streaming_mode eq 'parallel')
- {
- $parallel_message = join("", 'streamed transaction \d+ was ', $action, ' using an parallel apply worker');
-
- $node_subscriber->wait_for_log(qr/$parallel_message/, $offset);
- }
-}
-
# Encapsulate all the common test steps which are related to "streaming" parameter
# so the same code can be run both for the streaming=on and streaming=parallel
# cases.
@@ -34,9 +19,6 @@ sub test_streaming
my ($node_A, $node_B, $node_C, $appname_B, $appname_C, $streaming_mode) =
@_;
- my $offset_B = 0;
- my $offset_C = 0;
-
my $oldpid_B = $node_A->safe_psql(
'postgres', "
SELECT pid FROM pg_stat_replication
@@ -78,23 +60,6 @@ sub test_streaming
# Expect all data is replicated on subscriber(s) after the commit.
###############################
- # If "streaming" parameter is specified as "parallel", we need to check
- # that streamed transaction was prepared using an parallel apply worker.
- # We have to look for the DEBUG1 log messages about that, so bump up the
- # log verbosity.
- if ($streaming_mode eq 'parallel')
- {
- $node_B->append_conf('postgresql.conf', "log_min_messages = debug1");
- $node_B->reload;
-
- $node_C->append_conf('postgresql.conf', "log_min_messages = debug1");
- $node_C->reload;
- }
-
- # Check the subscriber log from now on.
- $offset_B = -s $node_B->logfile;
- $offset_C = -s $node_C->logfile;
-
# Insert, update and delete enough rows to exceed the 64kB limit.
# Then 2PC PREPARE
$node_A->safe_psql(
@@ -108,9 +73,6 @@ sub test_streaming
$node_A->wait_for_catchup($appname_B);
$node_B->wait_for_catchup($appname_C);
- check_parallel_log($node_B, $offset_B, $streaming_mode, "prepared");
- check_parallel_log($node_C, $offset_C, $streaming_mode, "prepared");
-
# check the transaction state is prepared on subscriber(s)
my $result =
$node_B->safe_psql('postgres',
@@ -164,10 +126,6 @@ sub test_streaming
# First, delete the data except for 2 rows (delete will be replicated)
$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;");
- # Check the subscriber log from now on.
- $offset_B = -s $node_B->logfile;
- $offset_C = -s $node_C->logfile;
-
# 2PC PREPARE with a nested ROLLBACK TO SAVEPOINT
$node_A->safe_psql(
'postgres', "
@@ -184,9 +142,6 @@ sub test_streaming
$node_A->wait_for_catchup($appname_B);
$node_B->wait_for_catchup($appname_C);
- check_parallel_log($node_B, $offset_B, $streaming_mode, "prepared");
- check_parallel_log($node_C, $offset_C, $streaming_mode, "prepared");
-
# check the transaction state prepared on subscriber(s)
$result =
$node_B->safe_psql('postgres',
@@ -233,16 +188,6 @@ sub test_streaming
$node_A->safe_psql('postgres', "DELETE FROM test_tab WHERE a > 2;");
$node_A->wait_for_catchup($appname_B);
$node_B->wait_for_catchup($appname_C);
-
- # Reset the log verbosity.
- if ($streaming_mode eq 'parallel')
- {
- $node_B->append_conf('postgresql.conf', "log_min_messages = warning");
- $node_B->reload;
-
- $node_C->append_conf('postgresql.conf', "log_min_messages = warning");
- $node_C->reload;
- }
}
###############################
diff --git a/src/test/subscription/t/023_twophase_stream.pl b/src/test/subscription/t/023_twophase_stream.pl
index 1fbc1c4..c49cb10 100644
--- a/src/test/subscription/t/023_twophase_stream.pl
+++ b/src/test/subscription/t/023_twophase_stream.pl
@@ -8,29 +8,12 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
-# Check the log that the streamed transaction was completed successfully
-# reported by parallel apply worker.
-sub check_parallel_log
-{
- my ($node_subscriber, $offset, $is_parallel, $action)= @_;
- my $parallel_message = '';
-
- if ($is_parallel)
- {
- $parallel_message = join("", 'streamed transaction \d+ was ', $action, ' using an parallel apply worker');
-
- $node_subscriber->wait_for_log(qr/$parallel_message/, $offset);
- }
-}
-
# Encapsulate all the common test steps which are related to "streaming"
# parameter so the same code can be run both for the streaming=on and
# streaming=parallel cases.
sub test_streaming
{
- my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
-
- my $offset = 0;
+ my ($node_publisher, $node_subscriber, $appname) = @_;
###############################
# Test 2PC PREPARE / COMMIT PREPARED.
@@ -40,20 +23,6 @@ sub test_streaming
# Expect all data is replicated on subscriber side after the commit.
###############################
- # If "streaming" parameter is specified as "parallel", we need to check
- # that streamed transaction was prepared using an parallel apply worker.
- # We have to look for the DEBUG1 log messages about that, so bump up the
- # log verbosity.
- if ($is_parallel)
- {
- $node_subscriber->append_conf('postgresql.conf',
- "log_min_messages = debug1");
- $node_subscriber->reload;
- }
-
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
-
# check that 2PC gets replicated to subscriber
# Insert, update and delete enough rows to exceed the 64kB limit.
$node_publisher->safe_psql(
@@ -66,8 +35,6 @@ sub test_streaming
$node_publisher->wait_for_catchup($appname);
- check_parallel_log($node_subscriber, $offset, $is_parallel, "prepared");
-
# check that transaction is in prepared state on subscriber
my $result = $node_subscriber->safe_psql('postgres',
"SELECT count(*) FROM pg_prepared_xacts;");
@@ -102,9 +69,6 @@ sub test_streaming
$node_publisher->safe_psql('postgres',
"DELETE FROM test_tab WHERE a > 2;");
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
-
# Then insert, update and delete enough rows to exceed the 64kB limit.
$node_publisher->safe_psql(
'postgres', q{
@@ -116,8 +80,6 @@ sub test_streaming
$node_publisher->wait_for_catchup($appname);
- check_parallel_log($node_subscriber, $offset, $is_parallel, "prepared");
-
# check that transaction is in prepared state on subscriber
$result = $node_subscriber->safe_psql('postgres',
"SELECT count(*) FROM pg_prepared_xacts;");
@@ -150,9 +112,6 @@ sub test_streaming
# Note: both publisher and subscriber do crash/restart.
###############################
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
-
$node_publisher->safe_psql(
'postgres', q{
BEGIN;
@@ -167,8 +126,6 @@ sub test_streaming
$node_publisher->start;
$node_subscriber->start;
- check_parallel_log($node_subscriber, $offset, $is_parallel, "prepared");
-
# commit post the restart
$node_publisher->safe_psql('postgres',
"COMMIT PREPARED 'test_prepared_tab';");
@@ -196,9 +153,6 @@ sub test_streaming
$node_publisher->safe_psql('postgres',
"DELETE FROM test_tab WHERE a > 2;");
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
-
# Then insert, update and delete enough rows to exceed the 64kB limit.
$node_publisher->safe_psql(
'postgres', q{
@@ -210,8 +164,6 @@ sub test_streaming
$node_publisher->wait_for_catchup($appname);
- check_parallel_log($node_subscriber, $offset, $is_parallel, "prepared");
-
# check that transaction is in prepared state on subscriber
$result = $node_subscriber->safe_psql('postgres',
"SELECT count(*) FROM pg_prepared_xacts;");
@@ -254,9 +206,6 @@ sub test_streaming
$node_publisher->safe_psql('postgres',
"DELETE FROM test_tab WHERE a > 2;");
- # Check the subscriber log from now on.
- $offset = -s $node_subscriber->logfile;
-
# Then insert, update and delete enough rows to exceed the 64kB limit.
$node_publisher->safe_psql(
'postgres', q{
@@ -268,8 +217,6 @@ sub test_streaming
$node_publisher->wait_for_catchup($appname);
- check_parallel_log($node_subscriber, $offset, $is_parallel, "prepared");
-
# check that transaction is in prepared state on subscriber
$result = $node_subscriber->safe_psql('postgres',
"SELECT count(*) FROM pg_prepared_xacts;");
@@ -301,14 +248,6 @@ sub test_streaming
$node_publisher->safe_psql('postgres',
"DELETE FROM test_tab WHERE a > 2;");
$node_publisher->wait_for_catchup($appname);
-
- # Reset the log verbosity.
- if ($is_parallel)
- {
- $node_subscriber->append_conf('postgresql.conf',
- "log_min_messages = warning");
- $node_subscriber->reload;
- }
}
###############################
@@ -376,7 +315,7 @@ my $result = $node_subscriber->safe_psql('postgres',
"SELECT count(*), count(c), count(d = 999) FROM test_tab");
is($result, qq(2|2|2), 'check initial data was copied to subscriber');
-test_streaming($node_publisher, $node_subscriber, $appname, 0);
+test_streaming($node_publisher, $node_subscriber, $appname);
######################################
# Test using streaming mode 'parallel'
@@ -399,7 +338,7 @@ $node_publisher->poll_query_until('postgres',
or die
"Timed out while waiting for apply to restart after changing SUBSCRIPTION";
-test_streaming($node_publisher, $node_subscriber, $appname, 1);
+test_streaming($node_publisher, $node_subscriber, $appname);
###############################
# check all the cleanup
diff --git a/src/test/subscription/t/032_streaming_parallel_safety.pl b/src/test/subscription/t/032_streaming_parallel_safety.pl
index 24bd54f..77fec4e 100644
--- a/src/test/subscription/t/032_streaming_parallel_safety.pl
+++ b/src/test/subscription/t/032_streaming_parallel_safety.pl
@@ -81,7 +81,8 @@ my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
# ============================================================================
# It is not allowed that the unique column in the relation on the
# subscriber-side is not the unique column on the publisher-side. Check the
-# error reported by parallel worker in this case.
+# error reported by parallel worker in this case. And after retrying in
+# apply worker, we check if the data is replicated successfully.
# ============================================================================
# First we check the unique index on normal table.
@@ -106,6 +107,10 @@ $node_subscriber->wait_for_log(
qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab1" using subscription parameter streaming = parallel/,
$offset);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? duplicate key value violates unique constraint "idx_tab1"/,
+ $offset);
+
# Drop the unique index on the subscriber, now it works.
$node_subscriber->safe_psql('postgres', "DROP INDEX idx_tab1");
@@ -114,7 +119,7 @@ $node_publisher->wait_for_catchup($appname);
my $result =
$node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab1");
-is($result, qq(5001), 'data replicated to subscriber after dropping index');
+is($result, qq(5001), 'data replicated to subscriber after dropping unique index to retry apply');
# Clean up test data from the environment.
$node_publisher->safe_psql('postgres', "TRUNCATE TABLE test_tab1");
@@ -135,19 +140,21 @@ $node_subscriber->wait_for_log(
qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming = parallel/,
$offset);
-# Drop the unique index on the subscriber, now it works.
-$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx");
-
# Wait for this streaming transaction to be applied in the apply worker.
$node_publisher->wait_for_catchup($appname);
$result =
$node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned");
-is($result, qq(5000), 'data replicated to subscriber after dropping index');
+is($result, qq(5000), 'data replicated to subscriber after retrying because of unique index');
+
+# Drop the unique index on the subscriber.
+$node_subscriber->safe_psql('postgres', "DROP INDEX test_tab_b_partition_idx");
# ============================================================================
# Triggers which execute non-immutable function are not allowed on the
# subscriber side. Check the error reported by parallel worker in this case.
+# And after retrying in apply worker, we check if the data is replicated
+# successfully.
# ============================================================================
# First we check the trigger function on normal table.
@@ -168,17 +175,6 @@ CREATE TRIGGER tri_tab1_unsafe
BEFORE INSERT ON public.test_tab1
FOR EACH ROW EXECUTE PROCEDURE trigger_func_tab1_unsafe();
ALTER TABLE test_tab1 ENABLE REPLICA TRIGGER tri_tab1_unsafe;
-
-CREATE FUNCTION trigger_func_tab1_safe() RETURNS TRIGGER AS \$\$
- BEGIN
- RAISE NOTICE 'test for safe trigger function';
- RETURN NEW;
- END
-\$\$ language plpgsql;
-ALTER FUNCTION trigger_func_tab1_safe IMMUTABLE;
-CREATE TRIGGER tri_tab1_safe
-BEFORE INSERT ON public.test_tab1
-FOR EACH ROW EXECUTE PROCEDURE trigger_func_tab1_safe();
});
# Check the subscriber log from now on.
@@ -203,27 +199,25 @@ $node_subscriber->wait_for_log(
qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab1" using subscription parameter streaming = parallel/,
$offset);
-# Using trigger with immutable function, now it works.
-$node_subscriber->safe_psql(
- 'postgres', qq{
-ALTER TABLE test_tab1 ENABLE REPLICA TRIGGER tri_tab1_safe;
-DROP TRIGGER tri_tab1_unsafe ON public.test_tab1;
-});
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? duplicate key value violates unique constraint "idx_tab2"/,
+ $offset);
+
+# Drop the unique index on the subscriber, now it works.
+$node_subscriber->safe_psql('postgres', "DROP INDEX idx_tab2");
# Wait for this streaming transaction to be applied in the apply worker.
$node_publisher->wait_for_catchup($appname);
$result =
$node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab1");
-is($result, qq(2), 'data replicated to subscriber after using immutable expression');
+is($result, qq(2), 'data replicated to subscriber after retrying because of trigger');
# Clean up test data from the environment.
$node_subscriber->safe_psql(
'postgres', qq{
-DROP INDEX idx_tab2;
-DROP TRIGGER tri_tab1_safe ON public.test_tab1;
+DROP TRIGGER tri_tab1_unsafe ON public.test_tab1;
DROP function trigger_func_tab1_unsafe;
-DROP function trigger_func_tab1_safe;
});
$node_publisher->safe_psql(
'postgres', qq{
@@ -256,21 +250,22 @@ $node_subscriber->wait_for_log(
qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming = parallel/,
$offset);
-# Drop the trigger on the subscriber, now it works.
-$node_subscriber->safe_psql('postgres',
- "DROP TRIGGER insert_trig ON test_tab_partition");
-
# Wait for this streaming transaction to be applied in the apply worker.
$node_publisher->wait_for_catchup($appname);
$result =
$node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned");
-is($result, qq(0), 'data replicated to subscriber after dropping trigger');
+is($result, qq(0), 'data replicated to subscriber after retrying because of trigger');
+
+# Drop the trigger on the subscriber.
+$node_subscriber->safe_psql('postgres',
+ "DROP TRIGGER insert_trig ON test_tab_partition");
# ============================================================================
# It is not allowed that column default value expression contains a
# non-immutable function on the subscriber side. Check the error reported by
-# parallel worker in this case.
+# parallel worker in this case. And after retrying in apply worker, we check
+# if the data is replicated successfully.
# ============================================================================
# First we check the column default value expression on normal table.
@@ -308,17 +303,13 @@ $node_subscriber->wait_for_log(
qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab1" using subscription parameter streaming = parallel/,
$offset);
-# Alter default values to immutable expression, now it works.
-$node_subscriber->safe_psql('postgres',
- "ALTER TABLE test_tab1 ALTER COLUMN b SET DEFAULT 1");
-
# Wait for this streaming transaction to be applied in the apply worker.
$node_publisher->wait_for_catchup($appname);
$result =
$node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab1");
is($result, qq(5001),
- 'data replicated to subscriber after using immutable expression');
+ 'data replicated to subscriber after retrying because of column default value');
# Clean up test data from the environment.
$node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab1 DROP COLUMN b");
@@ -344,22 +335,23 @@ $node_subscriber->wait_for_log(
qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming = parallel/,
$offset);
-# Drop default value on the subscriber, now it works.
-$node_subscriber->safe_psql('postgres',
- "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT");
-
# Wait for this streaming transaction to be applied in the apply worker.
$node_publisher->wait_for_catchup($appname);
$result =
$node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned");
is($result, qq(5000),
- 'data replicated to subscriber after dropping default value expression');
+ 'data replicated to subscriber after retrying because of column default value');
+
+# Drop default value on the subscriber.
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE test_tab_partition ALTER COLUMN b DROP DEFAULT");
# ============================================================================
# It is not allowed that domain constraint expression contains a non-immutable
# function on the subscriber side. Check the error reported by parallel
-# worker in this case.
+# worker in this case. And after retrying in apply worker, we check if the
+# data is replicated successfully.
# ============================================================================
# Because the column type of the partition table must be the same as its parent
@@ -400,17 +392,13 @@ $node_subscriber->wait_for_log(
qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab1" using subscription parameter streaming = parallel/,
$offset);
-# Drop domain constraint expression, now it works.
-$node_subscriber->safe_psql('postgres',
- "ALTER DOMAIN tmp_domain DROP CONSTRAINT domain_check");
-
# Wait for this streaming transaction to be applied in the apply worker.
$node_publisher->wait_for_catchup($appname);
$result =
$node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab1");
is($result, qq(5001),
- 'data replicated to subscriber after using immutable expression'
+ 'data replicated to subscriber after retrying because of domain'
);
# Clean up test data from the environment.
@@ -424,9 +412,10 @@ TRUNCATE TABLE test_tab2;
$node_publisher->wait_for_catchup($appname);
# ============================================================================
-# It is not allowed that constraint expression contains a non-immutable function
-# on the subscriber side. Check the error reported by parallel worker in this
-# case.
+# It is not allowed that constraint expression contains a non-immutable
+# function on the subscriber side. Check the error reported by parallel
+# worker in this case. And after retrying in apply worker, we check if the
+# data is replicated successfully.
# ============================================================================
# First we check the constraint expression on normal table.
@@ -459,23 +448,16 @@ $node_subscriber->wait_for_log(
qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab1" using subscription parameter streaming = parallel/,
$offset);
-# Alter constraint expression to immutable expression, now it works.
-$node_subscriber->safe_psql(
- 'postgres', qq{
-ALTER TABLE test_tab1 DROP CONSTRAINT const_tab1_unsafe;
-ALTER TABLE test_tab1 ADD CONSTRAINT const_tab1_safe CHECK(a >= 0);
-});
-
# Wait for this streaming transaction to be applied in the apply worker.
$node_publisher->wait_for_catchup($appname);
$result =
$node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab1");
is($result, qq(5001),
- 'data replicated to subscriber after using immutable expression');
+ 'data replicated to subscriber after retrying because of constraint');
# Clean up test data from the environment.
-$node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab1 DROP CONSTRAINT const_tab1_safe");
+$node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab1 DROP CONSTRAINT const_tab1_unsafe");
$node_publisher->safe_psql(
'postgres', qq{
TRUNCATE TABLE test_tab1;
@@ -498,21 +480,22 @@ $node_subscriber->wait_for_log(
qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming = parallel/,
$offset);
-# Drop constraint on the subscriber, now it works.
-$node_subscriber->safe_psql('postgres',
- "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con");
-
# Wait for this streaming transaction to be applied in the apply worker.
$node_publisher->wait_for_catchup($appname);
$result =
$node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned");
is($result, qq(0),
- 'data replicated to subscriber after dropping constraint expression');
+ 'data replicated to subscriber after retrying because of constraint');
+
+# Drop constraint on the subscriber.
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_con");
# ============================================================================
# It is not allowed that foreign key on the subscriber side. Check the error
-# reported by parallel worker in this case.
+# reported by parallel worker in this case. And after retrying in apply
+# worker, we check if the data is replicated successfully.
# ============================================================================
# First we check the foreign key on normal table.
@@ -553,8 +536,8 @@ $node_subscriber->wait_for_log(
qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab1" using subscription parameter streaming = parallel/,
$offset);
-# Drop the foreign key constraint on the subscriber, now it works.
-$node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab1 DROP CONSTRAINT test_tab1fk");
+# Insert dependent data on the publisher, now it works.
+$node_subscriber->safe_psql('postgres', "INSERT INTO test_tab2 VALUES(1)");
# Wait for this streaming transaction to be applied in the apply worker.
$node_publisher->wait_for_catchup($appname);
@@ -562,9 +545,10 @@ $node_publisher->wait_for_catchup($appname);
$result =
$node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab1");
is($result, qq(1),
- 'data replicated to subscriber after dropping the foreign key');
+ 'data replicated to subscriber after retrying because of foreign key');
# Clean up test data from the environment.
+$node_subscriber->safe_psql('postgres', "ALTER TABLE test_tab1 DROP CONSTRAINT test_tab1fk");
$node_publisher->safe_psql(
'postgres', qq{
TRUNCATE TABLE test_tab1;
@@ -590,17 +574,17 @@ $node_subscriber->wait_for_log(
qr/ERROR: ( [A-Z0-9]+:)? cannot replicate target relation "public.test_tab_partitioned" using subscription parameter streaming = parallel/,
$offset);
-# Drop the foreign key constraint on the subscriber, now it works.
-$node_subscriber->safe_psql('postgres',
- "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk");
-
# Wait for this streaming transaction to be applied in the apply worker.
$node_publisher->wait_for_catchup($appname);
$result =
$node_subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tab_partitioned");
is($result, qq(5000),
- 'data replicated to subscriber after dropping the foreign key');
+ 'data replicated to subscribers after retrying because of foreign key');
+
+# Drop the foreign key constraint on the subscriber.
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE test_tab_partition DROP CONSTRAINT test_tab_patition_fk");
$node_subscriber->stop;
$node_publisher->stop;
--
2.7.2.windows.1
view thread (625+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: RE: Perform streaming logical transactions by background workers and parallel apply
In-Reply-To: <OS0PR01MB5716546863340CA19EC5F8C394789@OS0PR01MB5716.jpnprd01.prod.outlook.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox